[165307.666469] lowmem_reserve[]: 0 0 13040 13040 [165307.666472] Node 0 Normal free:36816kB min:37012kB low:46264kB high:55516kB active_anon:80kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:284kB mapped:1524kB shmem:0kB slab_reclaimable:26392kB slab_unreclaimable:1101520kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165307.666473] lowmem_reserve[]: 0 0 0 0 [165307.666477] Node 1 Normal free:45124kB min:45728kB low:57160kB high:68592kB active_anon:444kB inactive_anon:576kB active_file:496kB inactive_file:1280kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:692kB mapped:4516kB shmem:0kB slab_reclaimable:33032kB slab_unreclaimable:958628kB kernel_stack:17328kB pagetables:3708kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165307.666477] lowmem_reserve[]: 0 0 0 0 [165307.666482] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165307.666486] Node 0 DMA32: 92*4kB (UEM) 2347*8kB (UEM) 1297*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59672kB [165307.666491] Node 0 Normal: 163*4kB (UEM) 129*8kB (UEM) 207*16kB (UEM) 290*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36996kB [165307.666495] Node 1 Normal: 298*4kB (UEM) 300*8kB (UEM) 207*16kB (UEM) 452*32kB (UM) 313*64kB (UEM) 39*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46392kB [165307.666496] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165307.666496] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165307.666497] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165307.666497] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165307.666498] 2322 total pagecache pages [165307.666498] 225 pages in swap cache [165307.666498] Swap cache stats: add 296736, delete 296511, find 88402/142157 [165307.666499] Free swap = 16248252kB [165307.666499] Total swap = 16319484kB [165307.666500] 8369063 pages RAM [165307.666500] 0 pages HighMem/MovableOnly [165307.666500] 241261 pages reserved [165308.299094] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165308.299096] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165308.299097] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165308.299103] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165308.299104] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165308.299105] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165308.299106] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165308.299107] Call Trace: [165308.299110] [] dump_stack+0x19/0x1b [165308.299113] [] warn_alloc_failed+0x110/0x180 [165308.299114] [] __alloc_pages_slowpath+0x6b6/0x724 [165308.299116] [] __alloc_pages_nodemask+0x405/0x420 [165308.299118] [] dma_generic_alloc_coherent+0x8f/0x140 [165308.299119] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165308.299130] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165308.299135] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165308.299141] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165308.299143] [] ? dma_pool_free+0xa7/0xd0 [165308.299148] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165308.299152] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165308.299156] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165308.299162] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165308.299164] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165308.299169] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165308.299173] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165308.299175] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165308.299177] [] process_one_work+0x17a/0x440 [165308.299178] [] worker_thread+0x126/0x3c0 [165308.299179] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165308.299181] [] kthread+0xcf/0xe0 [165308.299182] [] ? insert_kthread_work+0x40/0x40 [165308.299183] [] ret_from_fork+0x58/0x90 [165308.299184] [] ? insert_kthread_work+0x40/0x40 [165308.299185] Mem-Info: [165308.299189] active_anon:128 inactive_anon:13 isolated_anon:0 active_file:124 inactive_file:7 isolated_file:0 unevictable:6774 dirty:0 writeback:304 unstable:0 slab_reclaimable:15911 slab_unreclaimable:564252 mapped:1601 shmem:0 pagetables:1551 bounce:0 free:39682 free_pcp:0 free_cma:0 [165308.299193] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165308.299195] lowmem_reserve[]: 0 2580 15620 15620 [165308.299198] Node 0 DMA32 free:59664kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:24kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:68kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165308.299199] lowmem_reserve[]: 0 0 13040 13040 [165308.299203] Node 0 Normal free:37300kB min:37012kB low:46264kB high:55516kB active_anon:80kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:284kB mapped:1524kB shmem:0kB slab_reclaimable:26392kB slab_unreclaimable:1101520kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165308.299203] lowmem_reserve[]: 0 0 0 0 [165308.299207] Node 1 Normal free:46184kB min:45728kB low:57160kB high:68592kB active_anon:428kB inactive_anon:140kB active_file:496kB inactive_file:524kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:864kB mapped:4540kB shmem:0kB slab_reclaimable:33032kB slab_unreclaimable:958628kB kernel_stack:17328kB pagetables:3708kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165308.299208] lowmem_reserve[]: 0 0 0 0 [165308.299213] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165308.299217] Node 0 DMA32: 91*4kB (UEM) 2346*8kB (UEM) 1296*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59676kB [165308.299222] Node 0 Normal: 260*4kB (UEM) 172*8kB (UEM) 209*16kB (UEM) 292*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37824kB [165308.299226] Node 1 Normal: 470*4kB (UEM) 352*8kB (UEM) 203*16kB (UEM) 452*32kB (UM) 313*64kB (UEM) 39*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47432kB [165308.299227] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165308.299227] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165308.299228] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165308.299228] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165308.299229] 2145 total pagecache pages [165308.299229] 200 pages in swap cache [165308.299230] Swap cache stats: add 296816, delete 296616, find 88409/142168 [165308.299230] Free swap = 16248008kB [165308.299231] Total swap = 16319484kB [165308.299231] 8369063 pages RAM [165308.299232] 0 pages HighMem/MovableOnly [165308.299232] 241261 pages reserved [165308.299282] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165308.299283] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165308.299284] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165308.299286] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165308.299287] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165308.299288] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165308.299289] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165308.299290] Call Trace: [165308.299291] [] dump_stack+0x19/0x1b [165308.299293] [] warn_alloc_failed+0x110/0x180 [165308.299294] [] ? drain_pages+0xb0/0xb0 [165308.299295] [] __alloc_pages_slowpath+0x6b6/0x724 [165308.299297] [] __alloc_pages_nodemask+0x405/0x420 [165308.299298] [] alloc_pages_current+0x98/0x110 [165308.299300] [] __get_free_pages+0xe/0x40 [165308.299302] [] swiotlb_alloc_coherent+0x5e/0x150 [165308.299304] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165308.299310] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165308.299315] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165308.299319] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165308.299321] [] ? dma_pool_free+0xa7/0xd0 [165308.299325] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165308.299329] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165308.299333] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165308.299338] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165308.299340] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165308.299344] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165308.299347] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165308.299349] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165308.299351] [] process_one_work+0x17a/0x440 [165308.299352] [] worker_thread+0x126/0x3c0 [165308.299353] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165308.299354] [] kthread+0xcf/0xe0 [165308.299355] [] ? insert_kthread_work+0x40/0x40 [165308.299357] [] ret_from_fork+0x58/0x90 [165308.299358] [] ? insert_kthread_work+0x40/0x40 [165308.299358] Mem-Info: [165308.299362] active_anon:128 inactive_anon:13 isolated_anon:0 active_file:124 inactive_file:7 isolated_file:0 unevictable:6774 dirty:0 writeback:304 unstable:0 slab_reclaimable:15911 slab_unreclaimable:564252 mapped:1601 shmem:0 pagetables:1551 bounce:0 free:39682 free_pcp:0 free_cma:0 [165308.299365] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165308.299366] lowmem_reserve[]: 0 2580 15620 15620 [165308.299370] Node 0 DMA32 free:59664kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:24kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:68kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165308.299371] lowmem_reserve[]: 0 0 13040 13040 [165308.299374] Node 0 Normal free:37300kB min:37012kB low:46264kB high:55516kB active_anon:80kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:284kB mapped:1524kB shmem:0kB slab_reclaimable:26392kB slab_unreclaimable:1101520kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165308.299375] lowmem_reserve[]: 0 0 0 0 [165308.299378] Node 1 Normal free:46184kB min:45728kB low:57160kB high:68592kB active_anon:428kB inactive_anon:140kB active_file:496kB inactive_file:524kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:864kB mapped:4540kB shmem:0kB slab_reclaimable:33032kB slab_unreclaimable:958628kB kernel_stack:17328kB pagetables:3708kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165308.299379] lowmem_reserve[]: 0 0 0 0 [165308.299384] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165308.299388] Node 0 DMA32: 91*4kB (UEM) 2346*8kB (UEM) 1296*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59676kB [165308.299393] Node 0 Normal: 260*4kB (UEM) 172*8kB (UEM) 209*16kB (UEM) 292*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37824kB [165308.299397] Node 1 Normal: 470*4kB (UEM) 352*8kB (UEM) 203*16kB (UEM) 452*32kB (UM) 313*64kB (UEM) 39*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47432kB [165308.299397] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165308.299398] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165308.299398] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165308.299399] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165308.299399] 2145 total pagecache pages [165308.299400] 200 pages in swap cache [165308.299400] Swap cache stats: add 296816, delete 296616, find 88409/142168 [165308.299400] Free swap = 16248008kB [165308.299401] Total swap = 16319484kB [165308.299401] 8369063 pages RAM [165308.299401] 0 pages HighMem/MovableOnly [165308.299402] 241261 pages reserved [165309.667786] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165309.667788] CPU: 6 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165309.667789] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165309.667793] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165309.667794] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165309.667795] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165309.667796] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165309.667797] Call Trace: [165309.667799] [] dump_stack+0x19/0x1b [165309.667801] [] warn_alloc_failed+0x110/0x180 [165309.667802] [] __alloc_pages_slowpath+0x6b6/0x724 [165309.667804] [] __alloc_pages_nodemask+0x405/0x420 [165309.667805] [] dma_generic_alloc_coherent+0x8f/0x140 [165309.667807] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165309.667816] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165309.667821] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165309.667827] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165309.667828] [] ? dma_pool_free+0xa7/0xd0 [165309.667833] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165309.667837] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165309.667841] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165309.667847] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165309.667850] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165309.667854] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165309.667858] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165309.667860] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165309.667861] [] process_one_work+0x17a/0x440 [165309.667863] [] worker_thread+0x126/0x3c0 [165309.667864] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165309.667865] [] kthread+0xcf/0xe0 [165309.667866] [] ? insert_kthread_work+0x40/0x40 [165309.667867] [] ret_from_fork+0x58/0x90 [165309.667868] [] ? insert_kthread_work+0x40/0x40 [165309.667869] Mem-Info: [165309.667873] active_anon:86 inactive_anon:103 isolated_anon:0 active_file:97 inactive_file:187 isolated_file:0 unevictable:6774 dirty:0 writeback:310 unstable:0 slab_reclaimable:15947 slab_unreclaimable:564252 mapped:1633 shmem:0 pagetables:1551 bounce:0 free:39402 free_pcp:0 free_cma:0 [165309.667877] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165309.667878] lowmem_reserve[]: 0 2580 15620 15620 [165309.667882] Node 0 DMA32 free:59164kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:28kB active_file:12kB inactive_file:4kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:72kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:63 all_unreclaimable? no [165309.667883] lowmem_reserve[]: 0 0 13040 13040 [165309.667886] Node 0 Normal free:36940kB min:37012kB low:46264kB high:55516kB active_anon:44kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:308kB mapped:1564kB shmem:0kB slab_reclaimable:26392kB slab_unreclaimable:1101520kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165309.667887] lowmem_reserve[]: 0 0 0 0 [165309.667890] Node 1 Normal free:45924kB min:45728kB low:57160kB high:68592kB active_anon:300kB inactive_anon:524kB active_file:528kB inactive_file:764kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:860kB mapped:4628kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958628kB kernel_stack:17328kB pagetables:3708kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165309.667891] lowmem_reserve[]: 0 0 0 0 [165309.667896] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165309.667901] Node 0 DMA32: 66*4kB (UEM) 2347*8kB (UEM) 1298*16kB (UEM) 460*32kB (UM) 74*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59392kB [165309.667905] Node 0 Normal: 252*4kB (UEM) 165*8kB (UEM) 209*16kB (UEM) 292*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37736kB [165309.667909] Node 1 Normal: 337*4kB (UEM) 333*8kB (UEM) 207*16kB (UEM) 452*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46684kB [165309.667910] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165309.667911] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165309.667911] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165309.667912] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165309.667912] 2150 total pagecache pages [165309.667913] 230 pages in swap cache [165309.667914] Swap cache stats: add 297034, delete 296804, find 88515/142335 [165309.667914] Free swap = 16248128kB [165309.667914] Total swap = 16319484kB [165309.667915] 8369063 pages RAM [165309.667915] 0 pages HighMem/MovableOnly [165309.667916] 241261 pages reserved [165309.667946] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165309.667948] CPU: 6 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165309.667948] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165309.667951] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165309.667952] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165309.667953] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165309.667954] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165309.667954] Call Trace: [165309.667956] [] dump_stack+0x19/0x1b [165309.667958] [] warn_alloc_failed+0x110/0x180 [165309.667959] [] ? drain_pages+0xb0/0xb0 [165309.667960] [] __alloc_pages_slowpath+0x6b6/0x724 [165309.667961] [] __alloc_pages_nodemask+0x405/0x420 [165309.667963] [] alloc_pages_current+0x98/0x110 [165309.667964] [] __get_free_pages+0xe/0x40 [165309.667966] [] swiotlb_alloc_coherent+0x5e/0x150 [165309.667968] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165309.667974] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165309.667979] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165309.667983] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165309.667985] [] ? dma_pool_free+0xa7/0xd0 [165309.667989] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165309.667993] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165309.667997] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165309.668002] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165309.668004] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165309.668008] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165309.668011] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165309.668013] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165309.668015] [] process_one_work+0x17a/0x440 [165309.668016] [] worker_thread+0x126/0x3c0 [165309.668017] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165309.668018] [] kthread+0xcf/0xe0 [165309.668019] [] ? insert_kthread_work+0x40/0x40 [165309.668021] [] ret_from_fork+0x58/0x90 [165309.668022] [] ? insert_kthread_work+0x40/0x40 [165309.668022] Mem-Info: [165309.668026] active_anon:86 inactive_anon:103 isolated_anon:0 active_file:97 inactive_file:187 isolated_file:0 unevictable:6774 dirty:0 writeback:310 unstable:0 slab_reclaimable:15947 slab_unreclaimable:564252 mapped:1633 shmem:0 pagetables:1551 bounce:0 free:39402 free_pcp:0 free_cma:0 [165309.668030] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165309.668031] lowmem_reserve[]: 0 2580 15620 15620 [165309.668034] Node 0 DMA32 free:59164kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:28kB active_file:12kB inactive_file:4kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:72kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:63 all_unreclaimable? no [165309.668035] lowmem_reserve[]: 0 0 13040 13040 [165309.668039] Node 0 Normal free:36940kB min:37012kB low:46264kB high:55516kB active_anon:44kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:308kB mapped:1564kB shmem:0kB slab_reclaimable:26392kB slab_unreclaimable:1101520kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165309.668040] lowmem_reserve[]: 0 0 0 0 [165309.668043] Node 1 Normal free:45924kB min:45728kB low:57160kB high:68592kB active_anon:300kB inactive_anon:524kB active_file:528kB inactive_file:764kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:860kB mapped:4628kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958628kB kernel_stack:17328kB pagetables:3708kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165309.668044] lowmem_reserve[]: 0 0 0 0 [165309.668049] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165309.668053] Node 0 DMA32: 66*4kB (UEM) 2347*8kB (UEM) 1298*16kB (UEM) 460*32kB (UM) 74*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59392kB [165309.668057] Node 0 Normal: 252*4kB (UEM) 165*8kB (UEM) 209*16kB (UEM) 292*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37736kB [165309.668061] Node 1 Normal: 337*4kB (UEM) 333*8kB (UEM) 207*16kB (UEM) 452*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46684kB [165309.668062] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165309.668063] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165309.668063] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165309.668064] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165309.668064] 2150 total pagecache pages [165309.668064] 230 pages in swap cache [165309.668065] Swap cache stats: add 297034, delete 296804, find 88515/142335 [165309.668065] Free swap = 16248128kB [165309.668066] Total swap = 16319484kB [165309.668066] 8369063 pages RAM [165309.668066] 0 pages HighMem/MovableOnly [165309.668067] 241261 pages reserved [165310.367547] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165310.367549] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165310.367549] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165310.367553] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165310.367555] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165310.367556] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165310.367557] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165310.367557] Call Trace: [165310.367560] [] dump_stack+0x19/0x1b [165310.367562] [] warn_alloc_failed+0x110/0x180 [165310.367563] [] __alloc_pages_slowpath+0x6b6/0x724 [165310.367564] [] __alloc_pages_nodemask+0x405/0x420 [165310.367566] [] dma_generic_alloc_coherent+0x8f/0x140 [165310.367568] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165310.367582] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165310.367588] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165310.367595] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165310.367597] [] ? dma_pool_free+0xa7/0xd0 [165310.367601] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165310.367605] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165310.367610] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165310.367616] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165310.367618] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165310.367621] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165310.367625] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165310.367627] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165310.367629] [] process_one_work+0x17a/0x440 [165310.367630] [] worker_thread+0x126/0x3c0 [165310.367631] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165310.367633] [] kthread+0xcf/0xe0 [165310.367634] [] ? insert_kthread_work+0x40/0x40 [165310.367635] [] ret_from_fork+0x58/0x90 [165310.367636] [] ? insert_kthread_work+0x40/0x40 [165310.367637] Mem-Info: [165310.367641] active_anon:79 inactive_anon:87 isolated_anon:0 active_file:8 inactive_file:63 isolated_file:0 unevictable:6774 dirty:0 writeback:300 unstable:0 slab_reclaimable:15947 slab_unreclaimable:564252 mapped:1547 shmem:0 pagetables:1551 bounce:0 free:40018 free_pcp:0 free_cma:0 [165310.367645] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165310.367646] lowmem_reserve[]: 0 2580 15620 15620 [165310.367650] Node 0 DMA32 free:59652kB min:7320kB low:9148kB high:10980kB active_anon:16kB inactive_anon:24kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:76kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165310.367651] lowmem_reserve[]: 0 0 13040 13040 [165310.367654] Node 0 Normal free:37496kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:328kB mapped:1512kB shmem:0kB slab_reclaimable:26392kB slab_unreclaimable:1101520kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165310.367655] lowmem_reserve[]: 0 0 0 0 [165310.367659] Node 1 Normal free:47344kB min:45728kB low:57160kB high:68592kB active_anon:300kB inactive_anon:444kB active_file:196kB inactive_file:380kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:796kB mapped:4336kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958628kB kernel_stack:17376kB pagetables:3708kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165310.367659] lowmem_reserve[]: 0 0 0 0 [165310.367665] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165310.367669] Node 0 DMA32: 95*4kB (UEM) 2350*8kB (UEM) 1300*16kB (UEM) 459*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59660kB [165310.367674] Node 0 Normal: 181*4kB (UEM) 169*8kB (UEM) 209*16kB (UEM) 291*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37452kB [165310.367678] Node 1 Normal: 523*4kB (UEM) 354*8kB (UEM) 211*16kB (UEM) 447*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47500kB [165310.367679] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165310.367679] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165310.367680] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165310.367680] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165310.367681] 1960 total pagecache pages [165310.367681] 220 pages in swap cache [165310.367682] Swap cache stats: add 297055, delete 296835, find 88516/142337 [165310.367682] Free swap = 16248060kB [165310.367683] Total swap = 16319484kB [165310.367683] 8369063 pages RAM [165310.367684] 0 pages HighMem/MovableOnly [165310.367684] 241261 pages reserved [165310.367797] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165310.367799] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165310.367799] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165310.367802] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165310.367803] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165310.367804] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165310.367805] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165310.367805] Call Trace: [165310.367807] [] dump_stack+0x19/0x1b [165310.367809] [] warn_alloc_failed+0x110/0x180 [165310.367810] [] ? drain_pages+0xb0/0xb0 [165310.367811] [] __alloc_pages_slowpath+0x6b6/0x724 [165310.367812] [] __alloc_pages_nodemask+0x405/0x420 [165310.367814] [] alloc_pages_current+0x98/0x110 [165310.367815] [] __get_free_pages+0xe/0x40 [165310.367817] [] swiotlb_alloc_coherent+0x5e/0x150 [165310.367818] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165310.367825] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165310.367830] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165310.367834] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165310.367836] [] ? dma_pool_free+0xa7/0xd0 [165310.367840] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165310.367844] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165310.367848] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165310.367854] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165310.367856] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165310.367859] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165310.367863] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165310.367865] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165310.367866] [] process_one_work+0x17a/0x440 [165310.367868] [] worker_thread+0x126/0x3c0 [165310.367869] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165310.367870] [] kthread+0xcf/0xe0 [165310.367871] [] ? insert_kthread_work+0x40/0x40 [165310.367872] [] ret_from_fork+0x58/0x90 [165310.367873] [] ? insert_kthread_work+0x40/0x40 [165310.367874] Mem-Info: [165310.367877] active_anon:79 inactive_anon:87 isolated_anon:0 active_file:8 inactive_file:63 isolated_file:0 unevictable:6774 dirty:0 writeback:300 unstable:0 slab_reclaimable:15947 slab_unreclaimable:564252 mapped:1547 shmem:0 pagetables:1551 bounce:0 free:40018 free_pcp:0 free_cma:0 [165310.367880] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165310.367882] lowmem_reserve[]: 0 2580 15620 15620 [165310.367885] Node 0 DMA32 free:59652kB min:7320kB low:9148kB high:10980kB active_anon:16kB inactive_anon:24kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:76kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165310.367886] lowmem_reserve[]: 0 0 13040 13040 [165310.367890] Node 0 Normal free:37496kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:328kB mapped:1512kB shmem:0kB slab_reclaimable:26392kB slab_unreclaimable:1101520kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165310.367890] lowmem_reserve[]: 0 0 0 0 [165310.367894] Node 1 Normal free:47344kB min:45728kB low:57160kB high:68592kB active_anon:300kB inactive_anon:444kB active_file:196kB inactive_file:380kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:796kB mapped:4336kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958628kB kernel_stack:17376kB pagetables:3708kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:23 all_unreclaimable? no [165310.367895] lowmem_reserve[]: 0 0 0 0 [165310.367900] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165310.367904] Node 0 DMA32: 95*4kB (UEM) 2350*8kB (UEM) 1300*16kB (UEM) 459*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59660kB [165310.367909] Node 0 Normal: 181*4kB (UEM) 169*8kB (UEM) 209*16kB (UEM) 291*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37452kB [165310.367913] Node 1 Normal: 523*4kB (UEM) 354*8kB (UEM) 211*16kB (UEM) 447*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47500kB [165310.367913] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165310.367914] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165310.367915] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165310.367915] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165310.367915] 1960 total pagecache pages [165310.367916] 220 pages in swap cache [165310.367916] Swap cache stats: add 297055, delete 296835, find 88516/142337 [165310.367917] Free swap = 16248060kB [165310.367917] Total swap = 16319484kB [165310.367918] 8369063 pages RAM [165310.367918] 0 pages HighMem/MovableOnly [165310.367918] 241261 pages reserved [165311.670188] warn_alloc_failed: 1 callbacks suppressed [165311.670189] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165311.670190] CPU: 5 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165311.670191] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165311.670196] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165311.670198] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165311.670199] ffff8807e687b850 ffffffff81188820 0000000000000000 ffff88043ffdb000 [165311.670200] 0000000000000008 00000000000080d0 ffff8807e687b850 00000000c340495d [165311.670200] Call Trace: [165311.670203] [] dump_stack+0x19/0x1b [165311.670205] [] warn_alloc_failed+0x110/0x180 [165311.670206] [] __alloc_pages_slowpath+0x6b6/0x724 [165311.670208] [] __alloc_pages_nodemask+0x405/0x420 [165311.670209] [] alloc_pages_current+0x98/0x110 [165311.670211] [] __get_free_pages+0xe/0x40 [165311.670213] [] swiotlb_alloc_coherent+0x5e/0x150 [165311.670214] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165311.670225] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165311.670230] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165311.670235] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165311.670237] [] ? dma_pool_free+0xa7/0xd0 [165311.670242] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165311.670246] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165311.670250] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165311.670257] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165311.670259] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165311.670264] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165311.670268] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165311.670270] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165311.670272] [] process_one_work+0x17a/0x440 [165311.670273] [] worker_thread+0x126/0x3c0 [165311.670275] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165311.670276] [] kthread+0xcf/0xe0 [165311.670277] [] ? insert_kthread_work+0x40/0x40 [165311.670278] [] ret_from_fork+0x58/0x90 [165311.670279] [] ? insert_kthread_work+0x40/0x40 [165311.670280] Mem-Info: [165311.670284] active_anon:106 inactive_anon:119 isolated_anon:0 active_file:42 inactive_file:293 isolated_file:0 unevictable:6774 dirty:0 writeback:310 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564260 mapped:1609 shmem:0 pagetables:1551 bounce:0 free:39546 free_pcp:0 free_cma:0 [165311.670287] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165311.670289] lowmem_reserve[]: 0 2580 15620 15620 [165311.670292] Node 0 DMA32 free:59608kB min:7320kB low:9148kB high:10980kB active_anon:20kB inactive_anon:20kB active_file:12kB inactive_file:44kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:76kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165311.670293] lowmem_reserve[]: 0 0 13040 13040 [165311.670297] Node 0 Normal free:36592kB min:37012kB low:46264kB high:55516kB active_anon:144kB inactive_anon:0kB active_file:0kB inactive_file:372kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:328kB mapped:1544kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101552kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1 all_unreclaimable? no [165311.670298] lowmem_reserve[]: 0 0 0 0 [165311.670301] Node 1 Normal free:46404kB min:45728kB low:57160kB high:68592kB active_anon:260kB inactive_anon:480kB active_file:328kB inactive_file:756kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:836kB mapped:4552kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958628kB kernel_stack:17376kB pagetables:3708kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:216 all_unreclaimable? no [165311.670302] lowmem_reserve[]: 0 0 0 0 [165311.670307] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165311.670312] Node 0 DMA32: 93*4kB (UEM) 2350*8kB (UEM) 1297*16kB (UEM) 462*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59636kB [165311.670316] Node 0 Normal: 211*4kB (UEM) 146*8kB (UEM) 197*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37164kB [165311.670320] Node 1 Normal: 267*4kB (EM) 330*8kB (UEM) 211*16kB (UEM) 447*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46284kB [165311.670321] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165311.670322] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165311.670322] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165311.670323] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165311.670323] 2291 total pagecache pages [165311.670324] 272 pages in swap cache [165311.670325] Swap cache stats: add 297360, delete 297088, find 88590/142477 [165311.670325] Free swap = 16248032kB [165311.670325] Total swap = 16319484kB [165311.670326] 8369063 pages RAM [165311.670326] 0 pages HighMem/MovableOnly [165311.670326] 241261 pages reserved [165312.665999] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165312.666000] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165312.666001] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165312.666005] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165312.666006] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165312.666007] ffff8807e687b8a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165312.666008] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165312.666009] Call Trace: [165312.666011] [] dump_stack+0x19/0x1b [165312.666013] [] warn_alloc_failed+0x110/0x180 [165312.666014] [] ? drain_pages+0xb0/0xb0 [165312.666015] [] __alloc_pages_slowpath+0x6b6/0x724 [165312.666017] [] __alloc_pages_nodemask+0x405/0x420 [165312.666019] [] dma_generic_alloc_coherent+0x8f/0x140 [165312.666021] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165312.666029] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165312.666034] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165312.666039] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165312.666044] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165312.666047] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165312.666052] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165312.666058] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165312.666060] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165312.666063] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165312.666067] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165312.666069] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165312.666071] [] process_one_work+0x17a/0x440 [165312.666072] [] worker_thread+0x126/0x3c0 [165312.666074] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165312.666075] [] kthread+0xcf/0xe0 [165312.666076] [] ? insert_kthread_work+0x40/0x40 [165312.666077] [] ret_from_fork+0x58/0x90 [165312.666078] [] ? insert_kthread_work+0x40/0x40 [165312.666079] Mem-Info: [165312.666083] active_anon:93 inactive_anon:125 isolated_anon:0 active_file:93 inactive_file:230 isolated_file:0 unevictable:6774 dirty:0 writeback:337 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564260 mapped:1605 shmem:0 pagetables:1552 bounce:0 free:39490 free_pcp:0 free_cma:0 [165312.666087] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165312.666088] lowmem_reserve[]: 0 2580 15620 15620 [165312.666091] Node 0 DMA32 free:59664kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:20kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:80kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165312.666093] lowmem_reserve[]: 0 0 13040 13040 [165312.666096] Node 0 Normal free:37120kB min:37012kB low:46264kB high:55516kB active_anon:76kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:332kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101552kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165312.666097] lowmem_reserve[]: 0 0 0 0 [165312.666100] Node 1 Normal free:45596kB min:45728kB low:57160kB high:68592kB active_anon:292kB inactive_anon:608kB active_file:448kB inactive_file:1132kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:936kB mapped:4572kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958628kB kernel_stack:17376kB pagetables:3712kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165312.666101] lowmem_reserve[]: 0 0 0 0 [165312.666107] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165312.666111] Node 0 DMA32: 96*4kB (UEM) 2350*8kB (UEM) 1300*16kB (UEM) 461*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59664kB [165312.666116] Node 0 Normal: 183*4kB (UEM) 166*8kB (UEM) 205*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37340kB [165312.666120] Node 1 Normal: 169*4kB (UEM) 308*8kB (UEM) 211*16kB (UEM) 447*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [165312.666121] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165312.666121] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165312.666122] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165312.666123] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165312.666123] 2295 total pagecache pages [165312.666124] 293 pages in swap cache [165312.666124] Swap cache stats: add 297537, delete 297244, find 88653/142583 [165312.666125] Free swap = 16248052kB [165312.666125] Total swap = 16319484kB [165312.666126] 8369063 pages RAM [165312.666126] 0 pages HighMem/MovableOnly [165312.666126] 241261 pages reserved [165312.666157] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165312.666159] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165312.666159] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165312.666161] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165312.666163] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165312.666164] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165312.666164] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165312.666165] Call Trace: [165312.666166] [] dump_stack+0x19/0x1b [165312.666168] [] warn_alloc_failed+0x110/0x180 [165312.666169] [] ? drain_pages+0xb0/0xb0 [165312.666170] [] __alloc_pages_slowpath+0x6b6/0x724 [165312.666172] [] __alloc_pages_nodemask+0x405/0x420 [165312.666173] [] alloc_pages_current+0x98/0x110 [165312.666175] [] __get_free_pages+0xe/0x40 [165312.666177] [] swiotlb_alloc_coherent+0x5e/0x150 [165312.666178] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165312.666184] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165312.666189] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165312.666194] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165312.666198] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165312.666202] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165312.666206] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165312.666211] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165312.666213] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165312.666216] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165312.666220] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165312.666222] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165312.666223] [] process_one_work+0x17a/0x440 [165312.666225] [] worker_thread+0x126/0x3c0 [165312.666226] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165312.666227] [] kthread+0xcf/0xe0 [165312.666228] [] ? insert_kthread_work+0x40/0x40 [165312.666229] [] ret_from_fork+0x58/0x90 [165312.666230] [] ? insert_kthread_work+0x40/0x40 [165312.666231] Mem-Info: [165312.666234] active_anon:93 inactive_anon:125 isolated_anon:0 active_file:93 inactive_file:230 isolated_file:0 unevictable:6774 dirty:0 writeback:337 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564260 mapped:1605 shmem:0 pagetables:1552 bounce:0 free:39490 free_pcp:0 free_cma:0 [165312.666238] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165312.666239] lowmem_reserve[]: 0 2580 15620 15620 [165312.666242] Node 0 DMA32 free:59664kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:20kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:80kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165312.666243] lowmem_reserve[]: 0 0 13040 13040 [165312.666247] Node 0 Normal free:37120kB min:37012kB low:46264kB high:55516kB active_anon:76kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:332kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101552kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165312.666248] lowmem_reserve[]: 0 0 0 0 [165312.666251] Node 1 Normal free:45596kB min:45728kB low:57160kB high:68592kB active_anon:292kB inactive_anon:608kB active_file:448kB inactive_file:1132kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:936kB mapped:4572kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958628kB kernel_stack:17376kB pagetables:3712kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165312.666252] lowmem_reserve[]: 0 0 0 0 [165312.666257] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165312.666261] Node 0 DMA32: 96*4kB (UEM) 2350*8kB (UEM) 1300*16kB (UEM) 461*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59664kB [165312.666266] Node 0 Normal: 183*4kB (UEM) 166*8kB (UEM) 205*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37340kB [165312.666269] Node 1 Normal: 169*4kB (UEM) 308*8kB (UEM) 211*16kB (UEM) 447*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [165312.666270] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165312.666271] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165312.666271] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165312.666272] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165312.666272] 2295 total pagecache pages [165312.666272] 293 pages in swap cache [165312.666273] Swap cache stats: add 297537, delete 297244, find 88653/142583 [165312.666273] Free swap = 16248052kB [165312.666274] Total swap = 16319484kB [165312.666274] 8369063 pages RAM [165312.666274] 0 pages HighMem/MovableOnly [165312.666275] 241261 pages reserved [165313.665979] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165313.665981] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165313.665981] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165313.665985] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165313.665987] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165313.665988] ffff8807e687b8a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165313.665989] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165313.665989] Call Trace: [165313.665991] [] dump_stack+0x19/0x1b [165313.665993] [] warn_alloc_failed+0x110/0x180 [165313.665994] [] ? drain_pages+0xb0/0xb0 [165313.665996] [] __alloc_pages_slowpath+0x6b6/0x724 [165313.665997] [] __alloc_pages_nodemask+0x405/0x420 [165313.665999] [] dma_generic_alloc_coherent+0x8f/0x140 [165313.666000] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165313.666008] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165313.666013] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165313.666018] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165313.666023] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165313.666027] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165313.666031] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165313.666037] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165313.666039] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165313.666043] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165313.666046] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165313.666049] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165313.666050] [] process_one_work+0x17a/0x440 [165313.666051] [] worker_thread+0x126/0x3c0 [165313.666053] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165313.666054] [] kthread+0xcf/0xe0 [165313.666055] [] ? insert_kthread_work+0x40/0x40 [165313.666056] [] ret_from_fork+0x58/0x90 [165313.666057] [] ? insert_kthread_work+0x40/0x40 [165313.666058] Mem-Info: [165313.666062] active_anon:139 inactive_anon:116 isolated_anon:0 active_file:50 inactive_file:116 isolated_file:0 unevictable:6774 dirty:0 writeback:330 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564264 mapped:1614 shmem:0 pagetables:1556 bounce:0 free:39510 free_pcp:0 free_cma:0 [165313.666066] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165313.666067] lowmem_reserve[]: 0 2580 15620 15620 [165313.666071] Node 0 DMA32 free:59652kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:20kB active_file:8kB inactive_file:20kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:80kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7 all_unreclaimable? no [165313.666072] lowmem_reserve[]: 0 0 13040 13040 [165313.666075] Node 0 Normal free:36816kB min:37012kB low:46264kB high:55516kB active_anon:80kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:332kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101552kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165313.666076] lowmem_reserve[]: 0 0 0 0 [165313.666080] Node 1 Normal free:45992kB min:45728kB low:57160kB high:68592kB active_anon:472kB inactive_anon:572kB active_file:284kB inactive_file:672kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:908kB mapped:4608kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165313.666081] lowmem_reserve[]: 0 0 0 0 [165313.666086] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165313.666091] Node 0 DMA32: 95*4kB (UEM) 2349*8kB (UEM) 1299*16kB (UEM) 462*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59668kB [165313.666095] Node 0 Normal: 205*4kB (UEM) 166*8kB (UEM) 205*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37428kB [165313.666099] Node 1 Normal: 254*4kB (UEM) 338*8kB (UEM) 211*16kB (UEM) 447*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46296kB [165313.666100] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165313.666101] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165313.666101] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165313.666102] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165313.666102] 2235 total pagecache pages [165313.666103] 286 pages in swap cache [165313.666103] Swap cache stats: add 297631, delete 297345, find 88680/142632 [165313.666104] Free swap = 16248104kB [165313.666104] Total swap = 16319484kB [165313.666105] 8369063 pages RAM [165313.666105] 0 pages HighMem/MovableOnly [165313.666105] 241261 pages reserved [165313.666153] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165313.666154] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165313.666155] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165313.666157] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165313.666158] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165313.666159] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165313.666160] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165313.666161] Call Trace: [165313.666162] [] dump_stack+0x19/0x1b [165313.666164] [] warn_alloc_failed+0x110/0x180 [165313.666165] [] ? drain_pages+0xb0/0xb0 [165313.666166] [] __alloc_pages_slowpath+0x6b6/0x724 [165313.666167] [] __alloc_pages_nodemask+0x405/0x420 [165313.666169] [] alloc_pages_current+0x98/0x110 [165313.666170] [] __get_free_pages+0xe/0x40 [165313.666172] [] swiotlb_alloc_coherent+0x5e/0x150 [165313.666173] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165313.666179] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165313.666184] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165313.666189] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165313.666193] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165313.666197] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165313.666201] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165313.666207] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165313.666209] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165313.666212] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165313.666216] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165313.666218] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165313.666219] [] process_one_work+0x17a/0x440 [165313.666220] [] worker_thread+0x126/0x3c0 [165313.666222] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165313.666223] [] kthread+0xcf/0xe0 [165313.666224] [] ? insert_kthread_work+0x40/0x40 [165313.666225] [] ret_from_fork+0x58/0x90 [165313.666226] [] ? insert_kthread_work+0x40/0x40 [165313.666227] Mem-Info: [165313.666230] active_anon:139 inactive_anon:116 isolated_anon:0 active_file:50 inactive_file:116 isolated_file:0 unevictable:6774 dirty:0 writeback:330 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564264 mapped:1614 shmem:0 pagetables:1556 bounce:0 free:39510 free_pcp:0 free_cma:0 [165313.666233] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165313.666234] lowmem_reserve[]: 0 2580 15620 15620 [165313.666238] Node 0 DMA32 free:59652kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:20kB active_file:8kB inactive_file:20kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:80kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7 all_unreclaimable? no [165313.666239] lowmem_reserve[]: 0 0 13040 13040 [165313.666242] Node 0 Normal free:36816kB min:37012kB low:46264kB high:55516kB active_anon:80kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:332kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101552kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165313.666243] lowmem_reserve[]: 0 0 0 0 [165313.666246] Node 1 Normal free:45992kB min:45728kB low:57160kB high:68592kB active_anon:472kB inactive_anon:572kB active_file:284kB inactive_file:672kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:908kB mapped:4608kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165313.666247] lowmem_reserve[]: 0 0 0 0 [165313.666252] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165313.666257] Node 0 DMA32: 95*4kB (UEM) 2349*8kB (UEM) 1299*16kB (UEM) 462*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59668kB [165313.666261] Node 0 Normal: 205*4kB (UEM) 166*8kB (UEM) 205*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37428kB [165313.666265] Node 1 Normal: 254*4kB (UEM) 338*8kB (UEM) 211*16kB (UEM) 447*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46296kB [165313.666266] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165313.666266] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165313.666267] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165313.666267] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165313.666268] 2235 total pagecache pages [165313.666268] 286 pages in swap cache [165313.666269] Swap cache stats: add 297631, delete 297345, find 88680/142632 [165313.666269] Free swap = 16248104kB [165313.666269] Total swap = 16319484kB [165313.666270] 8369063 pages RAM [165313.666270] 0 pages HighMem/MovableOnly [165313.666270] 241261 pages reserved [165314.666604] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165314.666605] CPU: 5 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165314.666606] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165314.666610] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165314.666611] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165314.666612] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165314.666613] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165314.666614] Call Trace: [165314.666616] [] dump_stack+0x19/0x1b [165314.666618] [] warn_alloc_failed+0x110/0x180 [165314.666619] [] __alloc_pages_slowpath+0x6b6/0x724 [165314.666621] [] __alloc_pages_nodemask+0x405/0x420 [165314.666623] [] dma_generic_alloc_coherent+0x8f/0x140 [165314.666624] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165314.666632] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165314.666637] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165314.666642] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165314.666644] [] ? dma_pool_free+0xa7/0xd0 [165314.666649] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165314.666652] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165314.666657] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165314.666663] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165314.666665] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165314.666669] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165314.666672] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165314.666675] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165314.666676] [] process_one_work+0x17a/0x440 [165314.666677] [] worker_thread+0x126/0x3c0 [165314.666679] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165314.666680] [] kthread+0xcf/0xe0 [165314.666681] [] ? insert_kthread_work+0x40/0x40 [165314.666682] [] ret_from_fork+0x58/0x90 [165314.666683] [] ? insert_kthread_work+0x40/0x40 [165314.666684] Mem-Info: [165314.666688] active_anon:90 inactive_anon:107 isolated_anon:0 active_file:35 inactive_file:82 isolated_file:0 unevictable:6774 dirty:0 writeback:321 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564272 mapped:1553 shmem:0 pagetables:1556 bounce:0 free:39533 free_pcp:0 free_cma:0 [165314.666692] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165314.666693] lowmem_reserve[]: 0 2580 15620 15620 [165314.666697] Node 0 DMA32 free:59656kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:20kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:80kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165314.666698] lowmem_reserve[]: 0 0 13040 13040 [165314.666701] Node 0 Normal free:36904kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:364kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101584kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165314.666702] lowmem_reserve[]: 0 0 0 0 [165314.666706] Node 1 Normal free:45992kB min:45728kB low:57160kB high:68592kB active_anon:356kB inactive_anon:528kB active_file:220kB inactive_file:540kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:840kB mapped:4364kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? no [165314.666707] lowmem_reserve[]: 0 0 0 0 [165314.666712] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165314.666717] Node 0 DMA32: 93*4kB (UEM) 2348*8kB (UEM) 1298*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59668kB [165314.666721] Node 0 Normal: 166*4kB (UEM) 163*8kB (UEM) 209*16kB (UEM) 289*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37280kB [165314.666725] Node 1 Normal: 256*4kB (UEM) 345*8kB (UEM) 210*16kB (UEM) 448*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46376kB [165314.666726] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165314.666727] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165314.666727] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165314.666728] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165314.666728] 2188 total pagecache pages [165314.666729] 250 pages in swap cache [165314.666729] Swap cache stats: add 297693, delete 297443, find 88692/142652 [165314.666730] Free swap = 16248048kB [165314.666730] Total swap = 16319484kB [165314.666731] 8369063 pages RAM [165314.666731] 0 pages HighMem/MovableOnly [165314.666731] 241261 pages reserved [165314.666805] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165314.666806] CPU: 5 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165314.666807] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165314.666810] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165314.666811] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165314.666812] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165314.666813] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165314.666813] Call Trace: [165314.666815] [] dump_stack+0x19/0x1b [165314.666817] [] warn_alloc_failed+0x110/0x180 [165314.666817] [] ? drain_pages+0xb0/0xb0 [165314.666819] [] __alloc_pages_slowpath+0x6b6/0x724 [165314.666820] [] __alloc_pages_nodemask+0x405/0x420 [165314.666821] [] alloc_pages_current+0x98/0x110 [165314.666823] [] __get_free_pages+0xe/0x40 [165314.666824] [] swiotlb_alloc_coherent+0x5e/0x150 [165314.666826] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165314.666832] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165314.666838] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165314.666842] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165314.666844] [] ? dma_pool_free+0xa7/0xd0 [165314.666848] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165314.666851] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165314.666856] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165314.666861] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165314.666863] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165314.666867] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165314.666870] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165314.666872] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165314.666874] [] process_one_work+0x17a/0x440 [165314.666875] [] worker_thread+0x126/0x3c0 [165314.666876] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165314.666877] [] kthread+0xcf/0xe0 [165314.666879] [] ? insert_kthread_work+0x40/0x40 [165314.666880] [] ret_from_fork+0x58/0x90 [165314.666881] [] ? insert_kthread_work+0x40/0x40 [165314.666881] Mem-Info: [165314.666885] active_anon:90 inactive_anon:107 isolated_anon:0 active_file:35 inactive_file:82 isolated_file:0 unevictable:6774 dirty:0 writeback:321 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564272 mapped:1553 shmem:0 pagetables:1556 bounce:0 free:39533 free_pcp:0 free_cma:0 [165314.666889] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165314.666890] lowmem_reserve[]: 0 2580 15620 15620 [165314.666893] Node 0 DMA32 free:59656kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:20kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:80kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165314.666894] lowmem_reserve[]: 0 0 13040 13040 [165314.666897] Node 0 Normal free:36904kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:364kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101584kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165314.666898] lowmem_reserve[]: 0 0 0 0 [165314.666902] Node 1 Normal free:45992kB min:45728kB low:57160kB high:68592kB active_anon:356kB inactive_anon:528kB active_file:220kB inactive_file:540kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:840kB mapped:4364kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165314.666903] lowmem_reserve[]: 0 0 0 0 [165314.666908] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165314.666912] Node 0 DMA32: 93*4kB (UEM) 2348*8kB (UEM) 1298*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59668kB [165314.666916] Node 0 Normal: 166*4kB (UEM) 163*8kB (UEM) 209*16kB (UEM) 289*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37280kB [165314.666921] Node 1 Normal: 263*4kB (UEM) 345*8kB (UEM) 210*16kB (UEM) 448*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46404kB [165314.666921] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165314.666922] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165314.666923] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165314.666923] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165314.666923] 2188 total pagecache pages [165314.666924] 243 pages in swap cache [165314.666924] Swap cache stats: add 297693, delete 297450, find 88692/142652 [165314.666925] Free swap = 16248048kB [165314.666925] Total swap = 16319484kB [165314.666926] 8369063 pages RAM [165314.666926] 0 pages HighMem/MovableOnly [165314.666926] 241261 pages reserved [165315.335471] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165315.335473] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165315.335474] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165315.335479] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165315.335480] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165315.335482] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165315.335483] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165315.335483] Call Trace: [165315.335486] [] dump_stack+0x19/0x1b [165315.335488] [] warn_alloc_failed+0x110/0x180 [165315.335490] [] __alloc_pages_slowpath+0x6b6/0x724 [165315.335491] [] __alloc_pages_nodemask+0x405/0x420 [165315.335493] [] dma_generic_alloc_coherent+0x8f/0x140 [165315.335495] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165315.335505] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165315.335510] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165315.335515] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165315.335518] [] ? dma_pool_free+0xa7/0xd0 [165315.335522] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165315.335526] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165315.335531] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165315.335537] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165315.335539] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165315.335544] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165315.335548] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165315.335550] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165315.335552] [] process_one_work+0x17a/0x440 [165315.335553] [] worker_thread+0x126/0x3c0 [165315.335555] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165315.335556] [] kthread+0xcf/0xe0 [165315.335557] [] ? insert_kthread_work+0x40/0x40 [165315.335558] [] ret_from_fork+0x58/0x90 [165315.335559] [] ? insert_kthread_work+0x40/0x40 [165315.335560] Mem-Info: [165315.335564] active_anon:102 inactive_anon:86 isolated_anon:0 active_file:48 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:157 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564272 mapped:1563 shmem:0 pagetables:1556 bounce:0 free:39706 free_pcp:0 free_cma:0 [165315.335568] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165315.335569] lowmem_reserve[]: 0 2580 15620 15620 [165315.335573] Node 0 DMA32 free:59616kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:0kB inactive_file:8kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:352kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? no [165315.335574] lowmem_reserve[]: 0 0 13040 13040 [165315.335578] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:364kB mapped:1516kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101584kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165315.335579] lowmem_reserve[]: 0 0 0 0 [165315.335582] Node 1 Normal free:46620kB min:45728kB low:57160kB high:68592kB active_anon:432kB inactive_anon:408kB active_file:376kB inactive_file:232kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:268kB mapped:4384kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165315.335583] lowmem_reserve[]: 0 0 0 0 [165315.335588] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165315.335593] Node 0 DMA32: 96*4kB (UEM) 2349*8kB (UEM) 1298*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59656kB [165315.335597] Node 0 Normal: 257*4kB (UEM) 173*8kB (UEM) 209*16kB (UEM) 289*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37724kB [165315.335601] Node 1 Normal: 425*4kB (UEM) 339*8kB (EM) 204*16kB (UEM) 448*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46908kB [165315.335602] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165315.335603] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165315.335603] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165315.335604] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165315.335604] 2121 total pagecache pages [165315.335605] 212 pages in swap cache [165315.335606] Swap cache stats: add 297715, delete 297503, find 88696/142660 [165315.335606] Free swap = 16248036kB [165315.335606] Total swap = 16319484kB [165315.335607] 8369063 pages RAM [165315.335607] 0 pages HighMem/MovableOnly [165315.335607] 241261 pages reserved [165315.335667] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165315.335668] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165315.335669] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165315.335671] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165315.335672] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165315.335673] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165315.335675] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165315.335675] Call Trace: [165315.335677] [] dump_stack+0x19/0x1b [165315.335678] [] warn_alloc_failed+0x110/0x180 [165315.335679] [] ? drain_pages+0xb0/0xb0 [165315.335681] [] __alloc_pages_slowpath+0x6b6/0x724 [165315.335682] [] __alloc_pages_nodemask+0x405/0x420 [165315.335684] [] alloc_pages_current+0x98/0x110 [165315.335685] [] __get_free_pages+0xe/0x40 [165315.335687] [] swiotlb_alloc_coherent+0x5e/0x150 [165315.335689] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165315.335695] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165315.335700] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165315.335704] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165315.335706] [] ? dma_pool_free+0xa7/0xd0 [165315.335710] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165315.335714] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165315.335718] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165315.335724] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165315.335726] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165315.335729] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165315.335733] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165315.335735] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165315.335736] [] process_one_work+0x17a/0x440 [165315.335737] [] worker_thread+0x126/0x3c0 [165315.335739] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165315.335740] [] kthread+0xcf/0xe0 [165315.335741] [] ? insert_kthread_work+0x40/0x40 [165315.335742] [] ret_from_fork+0x58/0x90 [165315.335743] [] ? insert_kthread_work+0x40/0x40 [165315.335744] Mem-Info: [165315.335747] active_anon:102 inactive_anon:86 isolated_anon:0 active_file:48 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:157 unstable:0 slab_reclaimable:15955 slab_unreclaimable:564272 mapped:1563 shmem:0 pagetables:1556 bounce:0 free:39706 free_pcp:0 free_cma:0 [165315.335751] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165315.335752] lowmem_reserve[]: 0 2580 15620 15620 [165315.335756] Node 0 DMA32 free:59616kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:0kB inactive_file:8kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:352kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? no [165315.335757] lowmem_reserve[]: 0 0 13040 13040 [165315.335760] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:364kB mapped:1516kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101584kB kernel_stack:15568kB pagetables:2264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165315.335761] lowmem_reserve[]: 0 0 0 0 [165315.335765] Node 1 Normal free:46620kB min:45728kB low:57160kB high:68592kB active_anon:432kB inactive_anon:408kB active_file:376kB inactive_file:232kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:268kB mapped:4384kB shmem:0kB slab_reclaimable:33176kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165315.335766] lowmem_reserve[]: 0 0 0 0 [165315.335771] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165315.335775] Node 0 DMA32: 96*4kB (UEM) 2349*8kB (UEM) 1298*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59656kB [165315.335779] Node 0 Normal: 257*4kB (UEM) 173*8kB (UEM) 209*16kB (UEM) 289*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37724kB [165315.335783] Node 1 Normal: 425*4kB (UEM) 339*8kB (EM) 204*16kB (UEM) 448*32kB (UM) 313*64kB (UEM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46908kB [165315.335784] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165315.335785] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165315.335785] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165315.335786] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165315.335786] 2121 total pagecache pages [165315.335787] 212 pages in swap cache [165315.335787] Swap cache stats: add 297715, delete 297503, find 88696/142660 [165315.335788] Free swap = 16248036kB [165315.335788] Total swap = 16319484kB [165315.335788] 8369063 pages RAM [165315.335789] 0 pages HighMem/MovableOnly [165315.335789] 241261 pages reserved [165316.665902] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165316.665904] CPU: 5 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165316.665904] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165316.665909] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165316.665911] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165316.665912] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165316.665913] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165316.665913] Call Trace: [165316.665916] [] dump_stack+0x19/0x1b [165316.665918] [] warn_alloc_failed+0x110/0x180 [165316.665919] [] __alloc_pages_slowpath+0x6b6/0x724 [165316.665921] [] __alloc_pages_nodemask+0x405/0x420 [165316.665923] [] dma_generic_alloc_coherent+0x8f/0x140 [165316.665924] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165316.665933] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165316.665938] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165316.665944] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165316.665946] [] ? dma_pool_free+0xa7/0xd0 [165316.665951] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165316.665954] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165316.665959] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165316.665965] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165316.665968] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165316.665972] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165316.665976] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165316.665978] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165316.665980] [] process_one_work+0x17a/0x440 [165316.665981] [] worker_thread+0x126/0x3c0 [165316.665982] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165316.665983] [] kthread+0xcf/0xe0 [165316.665985] [] ? insert_kthread_work+0x40/0x40 [165316.665986] [] ret_from_fork+0x58/0x90 [165316.665987] [] ? insert_kthread_work+0x40/0x40 [165316.665988] Mem-Info: [165316.665992] active_anon:110 inactive_anon:116 isolated_anon:0 active_file:129 inactive_file:48 isolated_file:0 unevictable:6774 dirty:0 writeback:174 unstable:0 slab_reclaimable:15963 slab_unreclaimable:564272 mapped:1633 shmem:7 pagetables:1557 bounce:0 free:39498 free_pcp:0 free_cma:0 [165316.665996] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165316.665997] lowmem_reserve[]: 0 2580 15620 15620 [165316.666000] Node 0 DMA32 free:59592kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:4kB inactive_file:32kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:4kB mapped:348kB shmem:4kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:232kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:19 all_unreclaimable? no [165316.666001] lowmem_reserve[]: 0 0 13040 13040 [165316.666005] Node 0 Normal free:36808kB min:37012kB low:46264kB high:55516kB active_anon:68kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:400kB mapped:1612kB shmem:20kB slab_reclaimable:26424kB slab_unreclaimable:1101584kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165316.666006] lowmem_reserve[]: 0 0 0 0 [165316.666009] Node 1 Normal free:46012kB min:45728kB low:57160kB high:68592kB active_anon:372kB inactive_anon:592kB active_file:568kB inactive_file:228kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:292kB mapped:4572kB shmem:4kB slab_reclaimable:33208kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165316.666010] lowmem_reserve[]: 0 0 0 0 [165316.666015] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165316.666020] Node 0 DMA32: 90*4kB (UEM) 2351*8kB (UEM) 1297*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59632kB [165316.666024] Node 0 Normal: 187*4kB (UEM) 171*8kB (UEM) 209*16kB (UEM) 289*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37428kB [165316.666028] Node 1 Normal: 182*4kB (UEM) 345*8kB (UEM) 211*16kB (UEM) 449*32kB (UEM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46064kB [165316.666029] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165316.666030] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165316.666031] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165316.666031] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165316.666032] 2261 total pagecache pages [165316.666032] 275 pages in swap cache [165316.666033] Swap cache stats: add 297959, delete 297684, find 88767/142780 [165316.666033] Free swap = 16247980kB [165316.666034] Total swap = 16319484kB [165316.666034] 8369063 pages RAM [165316.666035] 0 pages HighMem/MovableOnly [165316.666035] 241261 pages reserved [165318.666799] warn_alloc_failed: 3 callbacks suppressed [165318.666800] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165318.666801] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165318.666802] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165318.666806] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165318.666808] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165318.666809] ffff8804062038a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165318.666810] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165318.666810] Call Trace: [165318.666813] [] dump_stack+0x19/0x1b [165318.666815] [] warn_alloc_failed+0x110/0x180 [165318.666816] [] ? drain_pages+0xb0/0xb0 [165318.666817] [] __alloc_pages_slowpath+0x6b6/0x724 [165318.666819] [] __alloc_pages_nodemask+0x405/0x420 [165318.666820] [] dma_generic_alloc_coherent+0x8f/0x140 [165318.666822] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165318.666832] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165318.666837] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165318.666842] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165318.666847] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165318.666851] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165318.666855] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165318.666861] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165318.666864] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165318.666868] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165318.666871] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165318.666873] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165318.666875] [] process_one_work+0x17a/0x440 [165318.666876] [] worker_thread+0x126/0x3c0 [165318.666878] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165318.666879] [] kthread+0xcf/0xe0 [165318.666880] [] ? insert_kthread_work+0x40/0x40 [165318.666881] [] ret_from_fork+0x58/0x90 [165318.666882] [] ? insert_kthread_work+0x40/0x40 [165318.666883] Mem-Info: [165318.666887] active_anon:130 inactive_anon:158 isolated_anon:0 active_file:65 inactive_file:238 isolated_file:0 unevictable:6774 dirty:0 writeback:251 unstable:0 slab_reclaimable:15963 slab_unreclaimable:564280 mapped:1622 shmem:4 pagetables:1558 bounce:0 free:39544 free_pcp:0 free_cma:0 [165318.666891] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165318.666892] lowmem_reserve[]: 0 2580 15620 15620 [165318.666896] Node 0 DMA32 free:59520kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:36kB active_file:4kB inactive_file:28kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:16kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165318.666897] lowmem_reserve[]: 0 0 13040 13040 [165318.666900] Node 0 Normal free:36996kB min:37012kB low:46264kB high:55516kB active_anon:112kB inactive_anon:0kB active_file:0kB inactive_file:160kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:444kB mapped:1552kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101616kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165318.666901] lowmem_reserve[]: 0 0 0 0 [165318.666905] Node 1 Normal free:46080kB min:45728kB low:57160kB high:68592kB active_anon:400kB inactive_anon:716kB active_file:360kB inactive_file:764kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:544kB mapped:4596kB shmem:16kB slab_reclaimable:33208kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165318.666906] lowmem_reserve[]: 0 0 0 0 [165318.666911] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165318.666915] Node 0 DMA32: 63*4kB (UEM) 2349*8kB (UEM) 1298*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59524kB [165318.666920] Node 0 Normal: 248*4kB (UEM) 134*8kB (UEM) 199*16kB (UEM) 288*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37184kB [165318.666924] Node 1 Normal: 307*4kB (UEM) 290*8kB (UEM) 212*16kB (UEM) 450*32kB (UEM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46172kB [165318.666925] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165318.666925] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165318.666926] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165318.666927] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165318.666927] 2345 total pagecache pages [165318.666927] 333 pages in swap cache [165318.666928] Swap cache stats: add 298396, delete 298063, find 88919/143037 [165318.666928] Free swap = 16248032kB [165318.666929] Total swap = 16319484kB [165318.666929] 8369063 pages RAM [165318.666930] 0 pages HighMem/MovableOnly [165318.666930] 241261 pages reserved [165318.666963] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165318.666964] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165318.666964] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165318.666967] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165318.666968] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165318.666969] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165318.666970] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165318.666970] Call Trace: [165318.666972] [] dump_stack+0x19/0x1b [165318.666973] [] warn_alloc_failed+0x110/0x180 [165318.666974] [] ? drain_pages+0xb0/0xb0 [165318.666976] [] __alloc_pages_slowpath+0x6b6/0x724 [165318.666977] [] __alloc_pages_nodemask+0x405/0x420 [165318.666978] [] alloc_pages_current+0x98/0x110 [165318.666980] [] __get_free_pages+0xe/0x40 [165318.666982] [] swiotlb_alloc_coherent+0x5e/0x150 [165318.666983] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165318.666989] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165318.666994] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165318.666999] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165318.667003] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165318.667007] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165318.667012] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165318.667017] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165318.667019] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165318.667022] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165318.667026] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165318.667028] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165318.667030] [] process_one_work+0x17a/0x440 [165318.667031] [] worker_thread+0x126/0x3c0 [165318.667032] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165318.667033] [] kthread+0xcf/0xe0 [165318.667034] [] ? insert_kthread_work+0x40/0x40 [165318.667036] [] ret_from_fork+0x58/0x90 [165318.667037] [] ? insert_kthread_work+0x40/0x40 [165318.667037] Mem-Info: [165318.667041] active_anon:130 inactive_anon:158 isolated_anon:0 active_file:65 inactive_file:238 isolated_file:0 unevictable:6774 dirty:0 writeback:251 unstable:0 slab_reclaimable:15963 slab_unreclaimable:564280 mapped:1622 shmem:4 pagetables:1558 bounce:0 free:39544 free_pcp:0 free_cma:0 [165318.667044] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165318.667046] lowmem_reserve[]: 0 2580 15620 15620 [165318.667049] Node 0 DMA32 free:59520kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:36kB active_file:4kB inactive_file:28kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:16kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165318.667050] lowmem_reserve[]: 0 0 13040 13040 [165318.667054] Node 0 Normal free:36996kB min:37012kB low:46264kB high:55516kB active_anon:112kB inactive_anon:0kB active_file:0kB inactive_file:160kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:444kB mapped:1552kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101616kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165318.667055] lowmem_reserve[]: 0 0 0 0 [165318.667059] Node 1 Normal free:46080kB min:45728kB low:57160kB high:68592kB active_anon:400kB inactive_anon:716kB active_file:360kB inactive_file:764kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:544kB mapped:4596kB shmem:16kB slab_reclaimable:33208kB slab_unreclaimable:958644kB kernel_stack:17376kB pagetables:3728kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165318.667060] lowmem_reserve[]: 0 0 0 0 [165318.667065] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165318.667069] Node 0 DMA32: 63*4kB (UEM) 2349*8kB (UEM) 1298*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59524kB [165318.667073] Node 0 Normal: 248*4kB (UEM) 134*8kB (UEM) 199*16kB (UEM) 288*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37184kB [165318.667078] Node 1 Normal: 307*4kB (UEM) 290*8kB (UEM) 212*16kB (UEM) 450*32kB (UEM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46172kB [165318.667078] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165318.667079] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165318.667079] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165318.667080] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165318.667080] 2345 total pagecache pages [165318.667081] 333 pages in swap cache [165318.667081] Swap cache stats: add 298396, delete 298063, find 88919/143037 [165318.667081] Free swap = 16248032kB [165318.667082] Total swap = 16319484kB [165318.667082] 8369063 pages RAM [165318.667082] 0 pages HighMem/MovableOnly [165318.667083] 241261 pages reserved [165324.364426] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165324.364427] CPU: 6 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165324.364428] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165324.364432] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165324.364433] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165324.364434] ffff8804062038a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165324.364435] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165324.364436] Call Trace: [165324.364438] [] dump_stack+0x19/0x1b [165324.364441] [] warn_alloc_failed+0x110/0x180 [165324.364442] [] ? drain_pages+0xb0/0xb0 [165324.364443] [] __alloc_pages_slowpath+0x6b6/0x724 [165324.364445] [] __alloc_pages_nodemask+0x405/0x420 [165324.364446] [] dma_generic_alloc_coherent+0x8f/0x140 [165324.364448] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165324.364457] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165324.364462] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165324.364468] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165324.364470] [] ? dma_pool_free+0xa7/0xd0 [165324.364474] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165324.364478] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165324.364482] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165324.364489] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165324.364491] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165324.364495] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165324.364499] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165324.364501] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165324.364502] [] process_one_work+0x17a/0x440 [165324.364504] [] worker_thread+0x126/0x3c0 [165324.364505] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165324.364506] [] kthread+0xcf/0xe0 [165324.364507] [] ? insert_kthread_work+0x40/0x40 [165324.364509] [] ret_from_fork+0x58/0x90 [165324.364510] [] ? insert_kthread_work+0x40/0x40 [165324.364510] Mem-Info: [165324.364514] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:67 inactive_file:231 isolated_file:0 unevictable:6774 dirty:0 writeback:30 unstable:0 slab_reclaimable:15963 slab_unreclaimable:564284 mapped:1580 shmem:0 pagetables:1559 bounce:0 free:39444 free_pcp:0 free_cma:0 [165324.364518] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165324.364519] lowmem_reserve[]: 0 2580 15620 15620 [165324.364523] Node 0 DMA32 free:59492kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:8kB active_file:16kB inactive_file:12kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:27 all_unreclaimable? no [165324.364524] lowmem_reserve[]: 0 0 13040 13040 [165324.364528] Node 0 Normal free:36924kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:8kB mapped:1512kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101616kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:419 all_unreclaimable? no [165324.364529] lowmem_reserve[]: 0 0 0 0 [165324.364533] Node 1 Normal free:45780kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:276kB inactive_file:1212kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:100kB mapped:4468kB shmem:0kB slab_reclaimable:33208kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:158 all_unreclaimable? no [165324.364533] lowmem_reserve[]: 0 0 0 0 [165324.364539] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165324.364543] Node 0 DMA32: 72*4kB (UEM) 2347*8kB (UEM) 1296*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165324.364548] Node 0 Normal: 250*4kB (UEM) 174*8kB (UEM) 215*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37832kB [165324.364552] Node 1 Normal: 520*4kB (UEM) 368*8kB (UEM) 218*16kB (UEM) 449*32kB (UEM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47712kB [165324.364553] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165324.364553] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165324.364554] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165324.364554] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165324.364555] 1923 total pagecache pages [165324.364555] 31 pages in swap cache [165324.364556] Swap cache stats: add 298672, delete 298641, find 89008/143177 [165324.364556] Free swap = 16247896kB [165324.364557] Total swap = 16319484kB [165324.364557] 8369063 pages RAM [165324.364558] 0 pages HighMem/MovableOnly [165324.364558] 241261 pages reserved [165324.364630] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165324.364631] CPU: 6 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165324.364632] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165324.364634] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165324.364635] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165324.364636] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165324.364637] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165324.364638] Call Trace: [165324.364639] [] dump_stack+0x19/0x1b [165324.364641] [] warn_alloc_failed+0x110/0x180 [165324.364642] [] ? drain_pages+0xb0/0xb0 [165324.364643] [] __alloc_pages_slowpath+0x6b6/0x724 [165324.364645] [] __alloc_pages_nodemask+0x405/0x420 [165324.364646] [] alloc_pages_current+0x98/0x110 [165324.364648] [] __get_free_pages+0xe/0x40 [165324.364649] [] swiotlb_alloc_coherent+0x5e/0x150 [165324.364651] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165324.364657] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165324.364662] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165324.364666] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165324.364668] [] ? dma_pool_free+0xa7/0xd0 [165324.364672] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165324.364676] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165324.364680] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165324.364685] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165324.364688] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165324.364691] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165324.364694] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165324.364697] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165324.364698] [] process_one_work+0x17a/0x440 [165324.364699] [] worker_thread+0x126/0x3c0 [165324.364700] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165324.364702] [] kthread+0xcf/0xe0 [165324.364703] [] ? insert_kthread_work+0x40/0x40 [165324.364704] [] ret_from_fork+0x58/0x90 [165324.364705] [] ? insert_kthread_work+0x40/0x40 [165324.364705] Mem-Info: [165324.364709] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:67 inactive_file:231 isolated_file:0 unevictable:6774 dirty:0 writeback:30 unstable:0 slab_reclaimable:15963 slab_unreclaimable:564284 mapped:1580 shmem:0 pagetables:1559 bounce:0 free:39444 free_pcp:0 free_cma:0 [165324.364713] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165324.364714] lowmem_reserve[]: 0 2580 15620 15620 [165324.364717] Node 0 DMA32 free:59492kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:8kB active_file:16kB inactive_file:12kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:41 all_unreclaimable? no [165324.364718] lowmem_reserve[]: 0 0 13040 13040 [165324.364722] Node 0 Normal free:36924kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:8kB mapped:1512kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101616kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:529 all_unreclaimable? no [165324.364723] lowmem_reserve[]: 0 0 0 0 [165324.364726] Node 1 Normal free:45780kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:276kB inactive_file:1212kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:100kB mapped:4468kB shmem:0kB slab_reclaimable:33208kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:402 all_unreclaimable? no [165324.364727] lowmem_reserve[]: 0 0 0 0 [165324.364732] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165324.364736] Node 0 DMA32: 72*4kB (UEM) 2347*8kB (UEM) 1296*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165324.364741] Node 0 Normal: 250*4kB (UEM) 174*8kB (UEM) 215*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37832kB [165324.364745] Node 1 Normal: 520*4kB (UEM) 368*8kB (UEM) 218*16kB (UEM) 449*32kB (UEM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47712kB [165324.364746] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165324.364746] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165324.364747] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165324.364747] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165324.364747] 1923 total pagecache pages [165324.364748] 31 pages in swap cache [165324.364748] Swap cache stats: add 298672, delete 298641, find 89008/143177 [165324.364749] Free swap = 16247896kB [165324.364749] Total swap = 16319484kB [165324.364750] 8369063 pages RAM [165324.364750] 0 pages HighMem/MovableOnly [165324.364750] 241261 pages reserved [165325.372450] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165325.372452] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165325.372452] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165325.372457] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165325.372458] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165325.372459] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165325.372460] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165325.372460] Call Trace: [165325.372463] [] dump_stack+0x19/0x1b [165325.372465] [] warn_alloc_failed+0x110/0x180 [165325.372466] [] __alloc_pages_slowpath+0x6b6/0x724 [165325.372467] [] __alloc_pages_nodemask+0x405/0x420 [165325.372469] [] dma_generic_alloc_coherent+0x8f/0x140 [165325.372471] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165325.372479] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165325.372484] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165325.372489] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165325.372491] [] ? dma_pool_free+0xa7/0xd0 [165325.372496] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165325.372499] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165325.372504] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165325.372510] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165325.372512] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165325.372516] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165325.372520] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165325.372522] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165325.372524] [] process_one_work+0x17a/0x440 [165325.372525] [] worker_thread+0x126/0x3c0 [165325.372526] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165325.372527] [] kthread+0xcf/0xe0 [165325.372529] [] ? insert_kthread_work+0x40/0x40 [165325.372530] [] ret_from_fork+0x58/0x90 [165325.372531] [] ? insert_kthread_work+0x40/0x40 [165325.372531] Mem-Info: [165325.372535] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:7 inactive_file:68 isolated_file:0 unevictable:6774 dirty:0 writeback:34 unstable:0 slab_reclaimable:15963 slab_unreclaimable:564284 mapped:1549 shmem:0 pagetables:1559 bounce:0 free:40056 free_pcp:0 free_cma:0 [165325.372539] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165325.372540] lowmem_reserve[]: 0 2580 15620 15620 [165325.372544] Node 0 DMA32 free:59628kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:32kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165325.372545] lowmem_reserve[]: 0 0 13040 13040 [165325.372548] Node 0 Normal free:37232kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:8kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101616kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165325.372549] lowmem_reserve[]: 0 0 0 0 [165325.372553] Node 1 Normal free:47784kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:16kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:116kB mapped:4348kB shmem:0kB slab_reclaimable:33208kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1 all_unreclaimable? no [165325.372554] lowmem_reserve[]: 0 0 0 0 [165325.372559] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165325.372563] Node 0 DMA32: 90*4kB (UEM) 2343*8kB (UE) 1294*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59680kB [165325.372568] Node 0 Normal: 152*4kB (UEM) 171*8kB (UEM) 215*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37416kB [165325.372572] Node 1 Normal: 552*4kB (UEM) 380*8kB (UEM) 218*16kB (UEM) 449*32kB (UEM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47936kB [165325.372573] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165325.372573] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165325.372574] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165325.372574] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165325.372575] 1747 total pagecache pages [165325.372575] 59 pages in swap cache [165325.372576] Swap cache stats: add 298748, delete 298689, find 89015/143194 [165325.372576] Free swap = 16247900kB [165325.372577] Total swap = 16319484kB [165325.372577] 8369063 pages RAM [165325.372578] 0 pages HighMem/MovableOnly [165325.372578] 241261 pages reserved [165325.372806] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165325.372808] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165325.372808] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165325.372811] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165325.372812] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165325.372813] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165325.372814] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165325.372814] Call Trace: [165325.372816] [] dump_stack+0x19/0x1b [165325.372818] [] warn_alloc_failed+0x110/0x180 [165325.372819] [] ? drain_pages+0xb0/0xb0 [165325.372820] [] __alloc_pages_slowpath+0x6b6/0x724 [165325.372821] [] __alloc_pages_nodemask+0x405/0x420 [165325.372822] [] alloc_pages_current+0x98/0x110 [165325.372824] [] __get_free_pages+0xe/0x40 [165325.372826] [] swiotlb_alloc_coherent+0x5e/0x150 [165325.372827] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165325.372833] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165325.372838] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165325.372843] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165325.372845] [] ? dma_pool_free+0xa7/0xd0 [165325.372849] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165325.372852] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165325.372857] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165325.372862] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165325.372864] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165325.372868] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165325.372871] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165325.372873] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165325.372875] [] process_one_work+0x17a/0x440 [165325.372876] [] worker_thread+0x126/0x3c0 [165325.372877] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165325.372878] [] kthread+0xcf/0xe0 [165325.372880] [] ? insert_kthread_work+0x40/0x40 [165325.372881] [] ret_from_fork+0x58/0x90 [165325.372882] [] ? insert_kthread_work+0x40/0x40 [165325.372882] Mem-Info: [165325.372886] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:7 inactive_file:68 isolated_file:0 unevictable:6774 dirty:0 writeback:34 unstable:0 slab_reclaimable:15963 slab_unreclaimable:564284 mapped:1549 shmem:0 pagetables:1559 bounce:0 free:40056 free_pcp:0 free_cma:0 [165325.372890] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165325.372891] lowmem_reserve[]: 0 2580 15620 15620 [165325.372895] Node 0 DMA32 free:59628kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:32kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165325.372896] lowmem_reserve[]: 0 0 13040 13040 [165325.372899] Node 0 Normal free:37232kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:8kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101616kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165325.372900] lowmem_reserve[]: 0 0 0 0 [165325.372904] Node 1 Normal free:47784kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:16kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:116kB mapped:4348kB shmem:0kB slab_reclaimable:33208kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no [165325.372905] lowmem_reserve[]: 0 0 0 0 [165325.372910] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165325.372914] Node 0 DMA32: 90*4kB (UEM) 2343*8kB (UE) 1294*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59680kB [165325.372918] Node 0 Normal: 152*4kB (UEM) 171*8kB (UEM) 215*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37416kB [165325.372923] Node 1 Normal: 552*4kB (UEM) 380*8kB (UEM) 218*16kB (UEM) 449*32kB (UEM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47936kB [165325.372923] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165325.372924] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165325.372924] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165325.372925] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165325.372925] 1747 total pagecache pages [165325.372926] 59 pages in swap cache [165325.372926] Swap cache stats: add 298748, delete 298689, find 89015/143194 [165325.372927] Free swap = 16247900kB [165325.372927] Total swap = 16319484kB [165325.372927] 8369063 pages RAM [165325.372928] 0 pages HighMem/MovableOnly [165325.372928] 241261 pages reserved [165326.665648] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165326.665650] CPU: 5 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165326.665650] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165326.665657] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165326.665658] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165326.665659] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165326.665660] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165326.665660] Call Trace: [165326.665665] [] dump_stack+0x19/0x1b [165326.665667] [] warn_alloc_failed+0x110/0x180 [165326.665669] [] __alloc_pages_slowpath+0x6b6/0x724 [165326.665670] [] __alloc_pages_nodemask+0x405/0x420 [165326.665672] [] dma_generic_alloc_coherent+0x8f/0x140 [165326.665674] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165326.665688] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165326.665693] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165326.665699] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165326.665701] [] ? dma_pool_free+0xa7/0xd0 [165326.665706] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165326.665710] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165326.665714] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165326.665722] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165326.665724] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165326.665729] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165326.665733] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165326.665735] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165326.665737] [] process_one_work+0x17a/0x440 [165326.665738] [] worker_thread+0x126/0x3c0 [165326.665740] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165326.665741] [] kthread+0xcf/0xe0 [165326.665742] [] ? insert_kthread_work+0x40/0x40 [165326.665744] [] ret_from_fork+0x58/0x90 [165326.665745] [] ? insert_kthread_work+0x40/0x40 [165326.665745] Mem-Info: [165326.665750] active_anon:16 inactive_anon:0 isolated_anon:0 active_file:148 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:60 unstable:0 slab_reclaimable:15971 slab_unreclaimable:564284 mapped:1567 shmem:0 pagetables:1559 bounce:0 free:39670 free_pcp:0 free_cma:0 [165326.665754] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165326.665755] lowmem_reserve[]: 0 2580 15620 15620 [165326.665759] Node 0 DMA32 free:59672kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165326.665760] lowmem_reserve[]: 0 0 13040 13040 [165326.665763] Node 0 Normal free:36812kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:284kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:24kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101616kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165326.665764] lowmem_reserve[]: 0 0 0 0 [165326.665768] Node 1 Normal free:46616kB min:45728kB low:57160kB high:68592kB active_anon:64kB inactive_anon:0kB active_file:296kB inactive_file:528kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:204kB mapped:4420kB shmem:0kB slab_reclaimable:33240kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165326.665769] lowmem_reserve[]: 0 0 0 0 [165326.665774] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165326.665779] Node 0 DMA32: 93*4kB (UEM) 2345*8kB (UEM) 1296*16kB (UEM) 461*32kB (UM) 75*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59676kB [165326.665783] Node 0 Normal: 232*4kB (UEM) 175*8kB (UEM) 214*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37752kB [165326.665788] Node 1 Normal: 350*4kB (UEM) 260*8kB (UEM) 208*16kB (EM) 448*32kB (UM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45976kB [165326.665789] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165326.665789] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165326.665790] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165326.665791] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165326.665791] 1761 total pagecache pages [165326.665791] 143 pages in swap cache [165326.665792] Swap cache stats: add 299462, delete 299319, find 89224/143549 [165326.665793] Free swap = 16247968kB [165326.665793] Total swap = 16319484kB [165326.665794] 8369063 pages RAM [165326.665794] 0 pages HighMem/MovableOnly [165326.665794] 241261 pages reserved [165326.665898] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165326.665899] CPU: 5 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165326.665900] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165326.665902] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165326.665904] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165326.665905] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165326.665906] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165326.665906] Call Trace: [165326.665908] [] dump_stack+0x19/0x1b [165326.665909] [] warn_alloc_failed+0x110/0x180 [165326.665910] [] ? drain_pages+0xb0/0xb0 [165326.665911] [] __alloc_pages_slowpath+0x6b6/0x724 [165326.665913] [] __alloc_pages_nodemask+0x405/0x420 [165326.665915] [] alloc_pages_current+0x98/0x110 [165326.665916] [] __get_free_pages+0xe/0x40 [165326.665918] [] swiotlb_alloc_coherent+0x5e/0x150 [165326.665920] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165326.665926] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165326.665931] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165326.665936] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165326.665937] [] ? dma_pool_free+0xa7/0xd0 [165326.665941] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165326.665945] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165326.665949] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165326.665955] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165326.665957] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165326.665960] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165326.665964] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165326.665966] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165326.665967] [] process_one_work+0x17a/0x440 [165326.665968] [] worker_thread+0x126/0x3c0 [165326.665970] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165326.665971] [] kthread+0xcf/0xe0 [165326.665972] [] ? insert_kthread_work+0x40/0x40 [165326.665973] [] ret_from_fork+0x58/0x90 [165326.665974] [] ? insert_kthread_work+0x40/0x40 [165326.665975] Mem-Info: [165326.665978] active_anon:16 inactive_anon:0 isolated_anon:0 active_file:148 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:60 unstable:0 slab_reclaimable:15971 slab_unreclaimable:564284 mapped:1567 shmem:0 pagetables:1559 bounce:0 free:39670 free_pcp:0 free_cma:0 [165326.665982] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165326.665983] lowmem_reserve[]: 0 2580 15620 15620 [165326.665987] Node 0 DMA32 free:59672kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165326.665988] lowmem_reserve[]: 0 0 13040 13040 [165326.665991] Node 0 Normal free:36812kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:284kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:24kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101616kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165326.665992] lowmem_reserve[]: 0 0 0 0 [165326.665996] Node 1 Normal free:46616kB min:45728kB low:57160kB high:68592kB active_anon:64kB inactive_anon:0kB active_file:296kB inactive_file:528kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:204kB mapped:4420kB shmem:0kB slab_reclaimable:33240kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165326.665997] lowmem_reserve[]: 0 0 0 0 [165326.666002] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165326.666011] Node 0 DMA32: 93*4kB (UEM) 2345*8kB (UEM) 1296*16kB (UEM) 461*32kB (UM) 75*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59676kB [165326.666015] Node 0 Normal: 232*4kB (UEM) 175*8kB (UEM) 214*16kB (UEM) 290*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37752kB [165326.666020] Node 1 Normal: 350*4kB (UEM) 260*8kB (UEM) 208*16kB (EM) 448*32kB (UM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45976kB [165326.666021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165326.666021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165326.666022] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165326.666023] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165326.666023] 1761 total pagecache pages [165326.666023] 143 pages in swap cache [165326.666024] Swap cache stats: add 299462, delete 299319, find 89224/143549 [165326.666024] Free swap = 16247968kB [165326.666025] Total swap = 16319484kB [165326.666025] 8369063 pages RAM [165326.666025] 0 pages HighMem/MovableOnly [165326.666026] 241261 pages reserved [165327.665610] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165327.665613] CPU: 5 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165327.665613] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165327.665622] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165327.665624] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165327.665625] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165327.665626] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165327.665626] Call Trace: [165327.665632] [] dump_stack+0x19/0x1b [165327.665636] [] warn_alloc_failed+0x110/0x180 [165327.665637] [] __alloc_pages_slowpath+0x6b6/0x724 [165327.665639] [] __alloc_pages_nodemask+0x405/0x420 [165327.665642] [] dma_generic_alloc_coherent+0x8f/0x140 [165327.665644] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165327.665664] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165327.665669] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165327.665675] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165327.665678] [] ? dma_pool_free+0xa7/0xd0 [165327.665683] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165327.665686] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165327.665691] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165327.665698] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165327.665701] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165327.665707] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165327.665712] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165327.665714] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165327.665716] [] process_one_work+0x17a/0x440 [165327.665718] [] worker_thread+0x126/0x3c0 [165327.665719] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165327.665720] [] kthread+0xcf/0xe0 [165327.665721] [] ? insert_kthread_work+0x40/0x40 [165327.665723] [] ret_from_fork+0x58/0x90 [165327.665724] [] ? insert_kthread_work+0x40/0x40 [165327.665725] Mem-Info: [165327.665729] active_anon:25 inactive_anon:0 isolated_anon:0 active_file:103 inactive_file:43 isolated_file:0 unevictable:6774 dirty:0 writeback:77 unstable:0 slab_reclaimable:15971 slab_unreclaimable:564288 mapped:1589 shmem:0 pagetables:1559 bounce:0 free:39529 free_pcp:0 free_cma:0 [165327.665733] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165327.665735] lowmem_reserve[]: 0 2580 15620 15620 [165327.665738] Node 0 DMA32 free:59644kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:8kB inactive_file:4kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165327.665739] lowmem_reserve[]: 0 0 13040 13040 [165327.665744] Node 0 Normal free:36980kB min:37012kB low:46264kB high:55516kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:24kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101632kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165327.665744] lowmem_reserve[]: 0 0 0 0 [165327.665748] Node 1 Normal free:45912kB min:45728kB low:57160kB high:68592kB active_anon:92kB inactive_anon:0kB active_file:440kB inactive_file:464kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:272kB mapped:4508kB shmem:0kB slab_reclaimable:33240kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165327.665749] lowmem_reserve[]: 0 0 0 0 [165327.665755] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165327.665759] Node 0 DMA32: 94*4kB (UEM) 2346*8kB (UEM) 1297*16kB (UEM) 458*32kB (UM) 76*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59672kB [165327.665763] Node 0 Normal: 193*4kB (UEM) 163*8kB (UEM) 213*16kB (UEM) 291*32kB (UEM) 314*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37452kB [165327.665768] Node 1 Normal: 318*4kB (UEM) 335*8kB (EM) 216*16kB (UEM) 448*32kB (UM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46576kB [165327.665769] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165327.665769] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165327.665770] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165327.665770] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165327.665771] 2063 total pagecache pages [165327.665771] 99 pages in swap cache [165327.665772] Swap cache stats: add 299719, delete 299620, find 89290/143670 [165327.665773] Free swap = 16247960kB [165327.665773] Total swap = 16319484kB [165327.665774] 8369063 pages RAM [165327.665774] 0 pages HighMem/MovableOnly [165327.665774] 241261 pages reserved [165327.665898] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165327.665899] CPU: 5 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165327.665900] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165327.665902] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165327.665904] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165327.665905] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165327.665906] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165327.665906] Call Trace: [165327.665908] [] dump_stack+0x19/0x1b [165327.665909] [] warn_alloc_failed+0x110/0x180 [165327.665911] [] ? drain_pages+0xb0/0xb0 [165327.665912] [] __alloc_pages_slowpath+0x6b6/0x724 [165327.665913] [] __alloc_pages_nodemask+0x405/0x420 [165327.665915] [] alloc_pages_current+0x98/0x110 [165327.665917] [] __get_free_pages+0xe/0x40 [165327.665919] [] swiotlb_alloc_coherent+0x5e/0x150 [165327.665921] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165327.665927] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165327.665932] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165327.665937] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165327.665938] [] ? dma_pool_free+0xa7/0xd0 [165327.665943] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165327.665946] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165327.665951] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165327.665956] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165327.665958] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165327.665962] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165327.665965] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165327.665968] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165327.665974] [] process_one_work+0x17a/0x440 [165327.665976] [] worker_thread+0x126/0x3c0 [165327.665977] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165327.665978] [] kthread+0xcf/0xe0 [165327.665979] [] ? insert_kthread_work+0x40/0x40 [165327.665980] [] ret_from_fork+0x58/0x90 [165327.665981] [] ? insert_kthread_work+0x40/0x40 [165327.665982] Mem-Info: [165327.665986] active_anon:25 inactive_anon:0 isolated_anon:0 active_file:103 inactive_file:43 isolated_file:0 unevictable:6774 dirty:0 writeback:77 unstable:0 slab_reclaimable:15971 slab_unreclaimable:564288 mapped:1589 shmem:0 pagetables:1559 bounce:0 free:39529 free_pcp:0 free_cma:0 [165327.665989] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165327.665991] lowmem_reserve[]: 0 2580 15620 15620 [165327.665994] Node 0 DMA32 free:59644kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:8kB inactive_file:4kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165327.665995] lowmem_reserve[]: 0 0 13040 13040 [165327.665999] Node 0 Normal free:36980kB min:37012kB low:46264kB high:55516kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:24kB mapped:1508kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101632kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165327.666000] lowmem_reserve[]: 0 0 0 0 [165327.666004] Node 1 Normal free:45912kB min:45728kB low:57160kB high:68592kB active_anon:92kB inactive_anon:0kB active_file:440kB inactive_file:464kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:272kB mapped:4508kB shmem:0kB slab_reclaimable:33240kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165327.666005] lowmem_reserve[]: 0 0 0 0 [165327.666010] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165327.666014] Node 0 DMA32: 94*4kB (UEM) 2346*8kB (UEM) 1297*16kB (UEM) 458*32kB (UM) 76*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59672kB [165327.666018] Node 0 Normal: 193*4kB (UEM) 163*8kB (UEM) 213*16kB (UEM) 291*32kB (UEM) 314*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37452kB [165327.666023] Node 1 Normal: 318*4kB (UEM) 335*8kB (EM) 216*16kB (UEM) 448*32kB (UM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46576kB [165327.666024] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165327.666025] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165327.666025] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165327.666026] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165327.666026] 2063 total pagecache pages [165327.666027] 99 pages in swap cache [165327.666027] Swap cache stats: add 299719, delete 299620, find 89290/143670 [165327.666028] Free swap = 16247960kB [165327.666028] Total swap = 16319484kB [165327.666028] 8369063 pages RAM [165327.666029] 0 pages HighMem/MovableOnly [165327.666029] 241261 pages reserved [165328.332451] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165328.332453] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165328.332454] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165328.332458] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165328.332459] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165328.332460] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165328.332461] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165328.332461] Call Trace: [165328.332464] [] dump_stack+0x19/0x1b [165328.332467] [] warn_alloc_failed+0x110/0x180 [165328.332468] [] __alloc_pages_slowpath+0x6b6/0x724 [165328.332469] [] __alloc_pages_nodemask+0x405/0x420 [165328.332471] [] dma_generic_alloc_coherent+0x8f/0x140 [165328.332473] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165328.332483] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165328.332488] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165328.332494] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165328.332496] [] ? dma_pool_free+0xa7/0xd0 [165328.332500] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165328.332504] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165328.332509] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165328.332515] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165328.332517] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165328.332521] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165328.332525] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165328.332527] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165328.332529] [] process_one_work+0x17a/0x440 [165328.332530] [] worker_thread+0x126/0x3c0 [165328.332532] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165328.332533] [] kthread+0xcf/0xe0 [165328.332534] [] ? insert_kthread_work+0x40/0x40 [165328.332535] [] ret_from_fork+0x58/0x90 [165328.332536] [] ? insert_kthread_work+0x40/0x40 [165328.332537] Mem-Info: [165328.332542] active_anon:6 inactive_anon:2 isolated_anon:0 active_file:129 inactive_file:61 isolated_file:0 unevictable:6774 dirty:0 writeback:78 unstable:0 slab_reclaimable:15971 slab_unreclaimable:564288 mapped:1597 shmem:0 pagetables:1559 bounce:0 free:39801 free_pcp:0 free_cma:0 [165328.332546] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165328.332547] lowmem_reserve[]: 0 2580 15620 15620 [165328.332551] Node 0 DMA32 free:59664kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:8kB inactive_file:4kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165328.332552] lowmem_reserve[]: 0 0 13040 13040 [165328.332556] Node 0 Normal free:36984kB min:37012kB low:46264kB high:55516kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:24kB mapped:1520kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101632kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165328.332557] lowmem_reserve[]: 0 0 0 0 [165328.332561] Node 1 Normal free:46976kB min:45728kB low:57160kB high:68592kB active_anon:16kB inactive_anon:204kB active_file:544kB inactive_file:468kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:276kB mapped:4528kB shmem:0kB slab_reclaimable:33240kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165328.332562] lowmem_reserve[]: 0 0 0 0 [165328.332567] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165328.332572] Node 0 DMA32: 91*4kB (UEM) 2344*8kB (UEM) 1295*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59676kB [165328.332576] Node 0 Normal: 214*4kB (UEM) 164*8kB (UEM) 213*16kB (UEM) 290*32kB (UEM) 314*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37512kB [165328.332580] Node 1 Normal: 413*4kB (UEM) 351*8kB (EM) 216*16kB (UEM) 448*32kB (UM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47084kB [165328.332581] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165328.332582] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165328.332582] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165328.332583] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165328.332583] 1983 total pagecache pages [165328.332584] 119 pages in swap cache [165328.332585] Swap cache stats: add 299793, delete 299674, find 89298/143689 [165328.332585] Free swap = 16247908kB [165328.332585] Total swap = 16319484kB [165328.332586] 8369063 pages RAM [165328.332586] 0 pages HighMem/MovableOnly [165328.332587] 241261 pages reserved [165328.332688] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165328.332690] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165328.332690] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165328.332693] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165328.332694] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165328.332695] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165328.332696] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165328.332696] Call Trace: [165328.332698] [] dump_stack+0x19/0x1b [165328.332699] [] warn_alloc_failed+0x110/0x180 [165328.332701] [] ? drain_pages+0xb0/0xb0 [165328.332702] [] __alloc_pages_slowpath+0x6b6/0x724 [165328.332703] [] __alloc_pages_nodemask+0x405/0x420 [165328.332705] [] alloc_pages_current+0x98/0x110 [165328.332707] [] __get_free_pages+0xe/0x40 [165328.332708] [] swiotlb_alloc_coherent+0x5e/0x150 [165328.332710] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165328.332716] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165328.332721] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165328.332726] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165328.332727] [] ? dma_pool_free+0xa7/0xd0 [165328.332731] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165328.332735] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165328.332739] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165328.332745] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165328.332747] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165328.332750] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165328.332754] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165328.332756] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165328.332757] [] process_one_work+0x17a/0x440 [165328.332759] [] worker_thread+0x126/0x3c0 [165328.332760] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165328.332761] [] kthread+0xcf/0xe0 [165328.332762] [] ? insert_kthread_work+0x40/0x40 [165328.332763] [] ret_from_fork+0x58/0x90 [165328.332764] [] ? insert_kthread_work+0x40/0x40 [165328.332765] Mem-Info: [165328.332768] active_anon:6 inactive_anon:2 isolated_anon:0 active_file:129 inactive_file:61 isolated_file:0 unevictable:6774 dirty:0 writeback:78 unstable:0 slab_reclaimable:15971 slab_unreclaimable:564288 mapped:1597 shmem:0 pagetables:1559 bounce:0 free:39801 free_pcp:0 free_cma:0 [165328.332772] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165328.332773] lowmem_reserve[]: 0 2580 15620 15620 [165328.332777] Node 0 DMA32 free:59664kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:8kB inactive_file:4kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165328.332778] lowmem_reserve[]: 0 0 13040 13040 [165328.332782] Node 0 Normal free:36984kB min:37012kB low:46264kB high:55516kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:24kB mapped:1520kB shmem:0kB slab_reclaimable:26424kB slab_unreclaimable:1101632kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165328.332782] lowmem_reserve[]: 0 0 0 0 [165328.332786] Node 1 Normal free:46976kB min:45728kB low:57160kB high:68592kB active_anon:16kB inactive_anon:204kB active_file:544kB inactive_file:468kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:276kB mapped:4528kB shmem:0kB slab_reclaimable:33240kB slab_unreclaimable:958660kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no [165328.332787] lowmem_reserve[]: 0 0 0 0 [165328.332792] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165328.332797] Node 0 DMA32: 91*4kB (UEM) 2344*8kB (UEM) 1295*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59676kB [165328.332801] Node 0 Normal: 214*4kB (UEM) 164*8kB (UEM) 213*16kB (UEM) 290*32kB (UEM) 314*64kB (UEM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37512kB [165328.332805] Node 1 Normal: 413*4kB (UEM) 351*8kB (EM) 216*16kB (UEM) 448*32kB (UM) 312*64kB (UM) 38*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47084kB [165328.332806] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165328.332806] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165328.332807] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165328.332807] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165328.332808] 1983 total pagecache pages [165328.332808] 119 pages in swap cache [165328.332809] Swap cache stats: add 299793, delete 299674, find 89298/143689 [165328.332809] Free swap = 16247908kB [165328.332809] Total swap = 16319484kB [165328.332810] 8369063 pages RAM [165328.332810] 0 pages HighMem/MovableOnly [165328.332810] 241261 pages reserved [165330.330049] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165330.330050] CPU: 0 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165330.330051] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165330.330055] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165330.330057] 00000000000080d0 00000000fa2ca73a ffff88073f657810 ffffffff816a3e2d [165330.330058] ffff88073f6578a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165330.330059] ffffffffffffff00 000080d000000000 ffff88073f657870 00000000fa2ca73a [165330.330059] Call Trace: [165330.330062] [] dump_stack+0x19/0x1b [165330.330064] [] warn_alloc_failed+0x110/0x180 [165330.330065] [] ? drain_pages+0xb0/0xb0 [165330.330066] [] __alloc_pages_slowpath+0x6b6/0x724 [165330.330067] [] __alloc_pages_nodemask+0x405/0x420 [165330.330069] [] dma_generic_alloc_coherent+0x8f/0x140 [165330.330071] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165330.330079] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165330.330084] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165330.330090] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165330.330094] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165330.330098] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165330.330103] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165330.330108] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165330.330111] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165330.330114] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165330.330118] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165330.330120] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165330.330122] [] process_one_work+0x17a/0x440 [165330.330123] [] worker_thread+0x126/0x3c0 [165330.330124] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165330.330125] [] kthread+0xcf/0xe0 [165330.330127] [] ? insert_kthread_work+0x40/0x40 [165330.330128] [] ret_from_fork+0x58/0x90 [165330.330129] [] ? insert_kthread_work+0x40/0x40 [165330.330129] Mem-Info: [165330.330134] active_anon:26 inactive_anon:0 isolated_anon:0 active_file:111 inactive_file:96 isolated_file:0 unevictable:6774 dirty:0 writeback:53 unstable:0 slab_reclaimable:16003 slab_unreclaimable:564300 mapped:1589 shmem:0 pagetables:1559 bounce:0 free:39446 free_pcp:0 free_cma:0 [165330.330138] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165330.330139] lowmem_reserve[]: 0 2580 15620 15620 [165330.330142] Node 0 DMA32 free:59540kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:60kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:356kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? no [165330.330143] lowmem_reserve[]: 0 0 13040 13040 [165330.330147] Node 0 Normal free:36904kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:76kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:32kB mapped:1548kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101664kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2480 all_unreclaimable? yes [165330.330148] lowmem_reserve[]: 0 0 0 0 [165330.330151] Node 1 Normal free:45760kB min:45728kB low:57160kB high:68592kB active_anon:128kB inactive_anon:168kB active_file:308kB inactive_file:616kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:168kB mapped:4452kB shmem:0kB slab_reclaimable:33272kB slab_unreclaimable:958676kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:113 all_unreclaimable? no [165330.330152] lowmem_reserve[]: 0 0 0 0 [165330.330157] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165330.330162] Node 0 DMA32: 86*4kB (UEM) 2346*8kB (UEM) 1297*16kB (UEM) 458*32kB (UE) 77*64kB (UEM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59576kB [165330.330166] Node 0 Normal: 246*4kB (UEM) 174*8kB (UEM) 213*16kB (UEM) 290*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37656kB [165330.330171] Node 1 Normal: 490*4kB (UEM) 372*8kB (UEM) 217*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47544kB [165330.330171] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165330.330172] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165330.330173] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165330.330173] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165330.330173] 1947 total pagecache pages [165330.330174] 84 pages in swap cache [165330.330175] Swap cache stats: add 300430, delete 300346, find 89467/143989 [165330.330175] Free swap = 16247904kB [165330.330176] Total swap = 16319484kB [165330.330176] 8369063 pages RAM [165330.330176] 0 pages HighMem/MovableOnly [165330.330177] 241261 pages reserved [165330.334286] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165330.334287] CPU: 0 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165330.334288] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165330.334292] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165330.334293] 00000000000080d0 00000000fa2ca73a ffff88073f6577c0 ffffffff816a3e2d [165330.334294] ffff88073f657850 ffffffff81188820 0000000000000000 ffff88043ffdb000 [165330.334295] 0000000000000008 00000000000080d0 ffff88073f657850 00000000fa2ca73a [165330.334296] Call Trace: [165330.334298] [] dump_stack+0x19/0x1b [165330.334300] [] warn_alloc_failed+0x110/0x180 [165330.334301] [] __alloc_pages_slowpath+0x6b6/0x724 [165330.334302] [] __alloc_pages_nodemask+0x405/0x420 [165330.334304] [] alloc_pages_current+0x98/0x110 [165330.334305] [] __get_free_pages+0xe/0x40 [165330.334307] [] swiotlb_alloc_coherent+0x5e/0x150 [165330.334308] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165330.334317] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165330.334322] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165330.334327] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165330.334331] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165330.334335] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165330.334339] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165330.334345] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165330.334347] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165330.334351] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165330.334355] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165330.334357] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165330.334359] [] process_one_work+0x17a/0x440 [165330.334360] [] worker_thread+0x126/0x3c0 [165330.334361] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165330.334362] [] kthread+0xcf/0xe0 [165330.334364] [] ? insert_kthread_work+0x40/0x40 [165330.334365] [] ret_from_fork+0x58/0x90 [165330.334366] [] ? insert_kthread_work+0x40/0x40 [165330.334366] Mem-Info: [165330.334370] active_anon:26 inactive_anon:0 isolated_anon:0 active_file:111 inactive_file:96 isolated_file:0 unevictable:6774 dirty:0 writeback:53 unstable:0 slab_reclaimable:16003 slab_unreclaimable:564300 mapped:1589 shmem:0 pagetables:1559 bounce:0 free:39452 free_pcp:0 free_cma:0 [165330.334374] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165330.334375] lowmem_reserve[]: 0 2580 15620 15620 [165330.334379] Node 0 DMA32 free:59456kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:60kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:356kB shmem:0kB slab_reclaimable:4220kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165330.334380] lowmem_reserve[]: 0 0 13040 13040 [165330.334384] Node 0 Normal free:36904kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:76kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:32kB mapped:1548kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101664kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2488 all_unreclaimable? yes [165330.334385] lowmem_reserve[]: 0 0 0 0 [165330.334388] Node 1 Normal free:45868kB min:45728kB low:57160kB high:68592kB active_anon:128kB inactive_anon:168kB active_file:308kB inactive_file:616kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:168kB mapped:4452kB shmem:0kB slab_reclaimable:33272kB slab_unreclaimable:958676kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:55 all_unreclaimable? no [165330.334389] lowmem_reserve[]: 0 0 0 0 [165330.334394] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165330.334399] Node 0 DMA32: 87*4kB (UEM) 2345*8kB (UEM) 1295*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59540kB [165330.334403] Node 0 Normal: 246*4kB (UEM) 174*8kB (UEM) 213*16kB (UEM) 290*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37656kB [165330.334407] Node 1 Normal: 482*4kB (UEM) 372*8kB (UEM) 217*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47512kB [165330.334408] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165330.334408] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165330.334409] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165330.334409] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165330.334410] 1947 total pagecache pages [165330.334410] 91 pages in swap cache [165330.334411] Swap cache stats: add 300438, delete 300347, find 89467/143989 [165330.334411] Free swap = 16247904kB [165330.334412] Total swap = 16319484kB [165330.334412] 8369063 pages RAM [165330.334412] 0 pages HighMem/MovableOnly [165330.334413] 241261 pages reserved [165330.368362] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165330.368363] CPU: 0 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165330.368364] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165330.368368] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165330.368369] 00000000000080d0 00000000fa2ca73a ffff88073f657810 ffffffff816a3e2d [165330.368370] ffff88073f6578a0 ffffffff81188820 0000000000000000 00000000ffffffff [165330.368371] ffffffffffffff00 000080d000000000 ffff88073f657870 00000000fa2ca73a [165330.368372] Call Trace: [165330.368374] [] dump_stack+0x19/0x1b [165330.368376] [] warn_alloc_failed+0x110/0x180 [165330.368377] [] __alloc_pages_slowpath+0x6b6/0x724 [165330.368379] [] __alloc_pages_nodemask+0x405/0x420 [165330.368381] [] dma_generic_alloc_coherent+0x8f/0x140 [165330.368382] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165330.368390] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165330.368395] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165330.368400] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165330.368402] [] ? dma_pool_free+0xa7/0xd0 [165330.368407] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165330.368410] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165330.368415] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165330.368421] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165330.368423] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165330.368427] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165330.368431] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165330.368433] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165330.368434] [] process_one_work+0x17a/0x440 [165330.368436] [] worker_thread+0x126/0x3c0 [165330.368437] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165330.368438] [] kthread+0xcf/0xe0 [165330.368439] [] ? insert_kthread_work+0x40/0x40 [165330.368440] [] ret_from_fork+0x58/0x90 [165330.368441] [] ? insert_kthread_work+0x40/0x40 [165330.368442] Mem-Info: [165330.368446] active_anon:1 inactive_anon:14 isolated_anon:0 active_file:6 inactive_file:62 isolated_file:0 unevictable:6774 dirty:0 writeback:69 unstable:0 slab_reclaimable:16019 slab_unreclaimable:564300 mapped:1545 shmem:0 pagetables:1559 bounce:0 free:39998 free_pcp:0 free_cma:0 [165330.368450] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165330.368451] lowmem_reserve[]: 0 2580 15620 15620 [165330.368455] Node 0 DMA32 free:59576kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:12kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165330.368456] lowmem_reserve[]: 0 0 13040 13040 [165330.368460] Node 0 Normal free:37092kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:68kB mapped:1508kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101664kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165330.368461] lowmem_reserve[]: 0 0 0 0 [165330.368464] Node 1 Normal free:47744kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:296kB active_file:212kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:188kB mapped:4328kB shmem:0kB slab_reclaimable:33272kB slab_unreclaimable:958676kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165330.368465] lowmem_reserve[]: 0 0 0 0 [165330.368471] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165330.368475] Node 0 DMA32: 92*4kB (UEM) 2346*8kB (UEM) 1296*16kB (UEM) 459*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59616kB [165330.368479] Node 0 Normal: 164*4kB (UEM) 174*8kB (UEM) 213*16kB (UEM) 290*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37328kB [165330.368483] Node 1 Normal: 539*4kB (UEM) 380*8kB (UEM) 218*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47820kB [165330.368484] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165330.368485] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165330.368486] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165330.368486] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165330.368487] 1813 total pagecache pages [165330.368487] 89 pages in swap cache [165330.368488] Swap cache stats: add 300440, delete 300351, find 89467/143989 [165330.368488] Free swap = 16247896kB [165330.368488] Total swap = 16319484kB [165330.368489] 8369063 pages RAM [165330.368489] 0 pages HighMem/MovableOnly [165330.368490] 241261 pages reserved [165330.368688] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165330.368690] CPU: 0 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165330.368690] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165330.368693] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165330.368694] 00000000000080d0 00000000fa2ca73a ffff88073f6577c0 ffffffff816a3e2d [165330.368695] ffff88073f657850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165330.368696] ffffffffffffff00 000080d000000000 ffff88073f657820 00000000fa2ca73a [165330.368696] Call Trace: [165330.368698] [] dump_stack+0x19/0x1b [165330.368700] [] warn_alloc_failed+0x110/0x180 [165330.368701] [] ? drain_pages+0xb0/0xb0 [165330.368702] [] __alloc_pages_slowpath+0x6b6/0x724 [165330.368704] [] __alloc_pages_nodemask+0x405/0x420 [165330.368705] [] alloc_pages_current+0x98/0x110 [165330.368707] [] __get_free_pages+0xe/0x40 [165330.368708] [] swiotlb_alloc_coherent+0x5e/0x150 [165330.368710] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165330.368716] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165330.368721] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165330.368726] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165330.368727] [] ? dma_pool_free+0xa7/0xd0 [165330.368732] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165330.368735] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165330.368740] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165330.368745] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165330.368747] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165330.368750] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165330.368754] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165330.368756] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165330.368758] [] process_one_work+0x17a/0x440 [165330.368759] [] worker_thread+0x126/0x3c0 [165330.368760] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165330.368761] [] kthread+0xcf/0xe0 [165330.368762] [] ? insert_kthread_work+0x40/0x40 [165330.368763] [] ret_from_fork+0x58/0x90 [165330.368764] [] ? insert_kthread_work+0x40/0x40 [165330.368765] Mem-Info: [165330.368769] active_anon:1 inactive_anon:14 isolated_anon:0 active_file:6 inactive_file:62 isolated_file:0 unevictable:6774 dirty:0 writeback:69 unstable:0 slab_reclaimable:16019 slab_unreclaimable:564300 mapped:1545 shmem:0 pagetables:1559 bounce:0 free:39998 free_pcp:0 free_cma:0 [165330.368772] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165330.368774] lowmem_reserve[]: 0 2580 15620 15620 [165330.368777] Node 0 DMA32 free:59576kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:12kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165330.368778] lowmem_reserve[]: 0 0 13040 13040 [165330.368782] Node 0 Normal free:37092kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:68kB mapped:1508kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101664kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165330.368783] lowmem_reserve[]: 0 0 0 0 [165330.368787] Node 1 Normal free:47744kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:296kB active_file:212kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:188kB mapped:4328kB shmem:0kB slab_reclaimable:33272kB slab_unreclaimable:958676kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165330.368788] lowmem_reserve[]: 0 0 0 0 [165330.368792] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165330.368797] Node 0 DMA32: 92*4kB (UEM) 2346*8kB (UEM) 1296*16kB (UEM) 459*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59616kB [165330.368801] Node 0 Normal: 164*4kB (UEM) 174*8kB (UEM) 213*16kB (UEM) 290*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37328kB [165330.368805] Node 1 Normal: 539*4kB (UEM) 380*8kB (UEM) 218*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47820kB [165330.368806] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165330.368806] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165330.368807] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165330.368807] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165330.368808] 1813 total pagecache pages [165330.368808] 89 pages in swap cache [165330.368809] Swap cache stats: add 300440, delete 300351, find 89467/143989 [165330.368809] Free swap = 16247896kB [165330.368809] Total swap = 16319484kB [165330.368810] 8369063 pages RAM [165330.368810] 0 pages HighMem/MovableOnly [165330.368810] 241261 pages reserved [165331.665782] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165331.665784] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165331.665785] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165331.665793] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165331.665795] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165331.665797] ffff8807e15678a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165331.665799] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165331.665800] Call Trace: [165331.665805] [] dump_stack+0x19/0x1b [165331.665808] [] warn_alloc_failed+0x110/0x180 [165331.665810] [] ? drain_pages+0xb0/0xb0 [165331.665813] [] __alloc_pages_slowpath+0x6b6/0x724 [165331.665815] [] __alloc_pages_nodemask+0x405/0x420 [165331.665819] [] dma_generic_alloc_coherent+0x8f/0x140 [165331.665822] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165331.665839] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165331.665853] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165331.665864] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165331.665873] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165331.665880] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165331.665889] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165331.665900] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165331.665905] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165331.665912] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165331.665919] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165331.665923] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165331.665926] [] process_one_work+0x17a/0x440 [165331.665929] [] worker_thread+0x126/0x3c0 [165331.665931] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165331.665933] [] kthread+0xcf/0xe0 [165331.665936] [] ? insert_kthread_work+0x40/0x40 [165331.665938] [] ret_from_fork+0x58/0x90 [165331.665940] [] ? insert_kthread_work+0x40/0x40 [165331.665941] Mem-Info: [165331.665949] active_anon:9 inactive_anon:0 isolated_anon:0 active_file:39 inactive_file:154 isolated_file:0 unevictable:6774 dirty:0 writeback:94 unstable:0 slab_reclaimable:16019 slab_unreclaimable:564300 mapped:1563 shmem:0 pagetables:1559 bounce:0 free:39524 free_pcp:0 free_cma:0 [165331.665956] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165331.665958] lowmem_reserve[]: 0 2580 15620 15620 [165331.665966] Node 0 DMA32 free:59484kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:20kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165331.665968] lowmem_reserve[]: 0 0 13040 13040 [165331.665975] Node 0 Normal free:36720kB min:37012kB low:46264kB high:55516kB active_anon:12kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:68kB mapped:1508kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101664kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165331.665976] lowmem_reserve[]: 0 0 0 0 [165331.665983] Node 1 Normal free:46312kB min:45728kB low:57160kB high:68592kB active_anon:24kB inactive_anon:0kB active_file:344kB inactive_file:732kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:288kB mapped:4404kB shmem:0kB slab_reclaimable:33272kB slab_unreclaimable:958676kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165331.665985] lowmem_reserve[]: 0 0 0 0 [165331.665995] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165331.666004] Node 0 DMA32: 75*4kB (UEM) 2348*8kB (UEM) 1298*16kB (UEM) 461*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59532kB [165331.666012] Node 0 Normal: 246*4kB (UEM) 172*8kB (UEM) 213*16kB (UEM) 290*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37640kB [165331.666020] Node 1 Normal: 362*4kB (UEM) 260*8kB (UEM) 218*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46152kB [165331.666022] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165331.666023] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165331.666024] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165331.666025] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165331.666026] 1836 total pagecache pages [165331.666027] 139 pages in swap cache [165331.666028] Swap cache stats: add 301023, delete 300884, find 89631/144266 [165331.666029] Free swap = 16247968kB [165331.666029] Total swap = 16319484kB [165331.666031] 8369063 pages RAM [165331.666031] 0 pages HighMem/MovableOnly [165331.666032] 241261 pages reserved [165331.666201] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165331.666203] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165331.666204] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165331.666209] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165331.666212] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165331.666213] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165331.666215] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165331.666216] Call Trace: [165331.666219] [] dump_stack+0x19/0x1b [165331.666223] [] warn_alloc_failed+0x110/0x180 [165331.666225] [] ? drain_pages+0xb0/0xb0 [165331.666227] [] __alloc_pages_slowpath+0x6b6/0x724 [165331.666230] [] __alloc_pages_nodemask+0x405/0x420 [165331.666232] [] alloc_pages_current+0x98/0x110 [165331.666235] [] __get_free_pages+0xe/0x40 [165331.666238] [] swiotlb_alloc_coherent+0x5e/0x150 [165331.666241] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165331.666253] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165331.666262] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165331.666271] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165331.666280] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165331.666287] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165331.666296] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165331.666306] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165331.666310] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165331.666316] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165331.666324] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165331.666328] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165331.666330] [] process_one_work+0x17a/0x440 [165331.666333] [] worker_thread+0x126/0x3c0 [165331.666335] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165331.666337] [] kthread+0xcf/0xe0 [165331.666340] [] ? insert_kthread_work+0x40/0x40 [165331.666342] [] ret_from_fork+0x58/0x90 [165331.666344] [] ? insert_kthread_work+0x40/0x40 [165331.666345] Mem-Info: [165331.666352] active_anon:9 inactive_anon:0 isolated_anon:0 active_file:39 inactive_file:154 isolated_file:0 unevictable:6774 dirty:0 writeback:94 unstable:0 slab_reclaimable:16019 slab_unreclaimable:564300 mapped:1563 shmem:0 pagetables:1559 bounce:0 free:39524 free_pcp:0 free_cma:0 [165331.666359] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165331.666361] lowmem_reserve[]: 0 2580 15620 15620 [165331.666369] Node 0 DMA32 free:59484kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:20kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165331.666370] lowmem_reserve[]: 0 0 13040 13040 [165331.666378] Node 0 Normal free:36720kB min:37012kB low:46264kB high:55516kB active_anon:12kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:68kB mapped:1508kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101664kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165331.666379] lowmem_reserve[]: 0 0 0 0 [165331.666387] Node 1 Normal free:46312kB min:45728kB low:57160kB high:68592kB active_anon:24kB inactive_anon:0kB active_file:344kB inactive_file:732kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:288kB mapped:4404kB shmem:0kB slab_reclaimable:33272kB slab_unreclaimable:958676kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165331.666388] lowmem_reserve[]: 0 0 0 0 [165331.666398] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165331.666406] Node 0 DMA32: 75*4kB (UEM) 2348*8kB (UEM) 1298*16kB (UEM) 461*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59532kB [165331.666414] Node 0 Normal: 246*4kB (UEM) 172*8kB (UEM) 213*16kB (UEM) 290*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37640kB [165331.666422] Node 1 Normal: 362*4kB (UEM) 260*8kB (UEM) 218*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46152kB [165331.666424] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165331.666425] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165331.666426] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165331.666427] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165331.666428] 1836 total pagecache pages [165331.666428] 139 pages in swap cache [165331.666429] Swap cache stats: add 301023, delete 300884, find 89631/144266 [165331.666430] Free swap = 16247968kB [165331.666431] Total swap = 16319484kB [165331.666432] 8369063 pages RAM [165331.666432] 0 pages HighMem/MovableOnly [165331.666433] 241261 pages reserved [165333.282170] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165333.282172] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165333.282173] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165333.282176] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165333.282178] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165333.282179] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165333.282180] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165333.282180] Call Trace: [165333.282183] [] dump_stack+0x19/0x1b [165333.282185] [] warn_alloc_failed+0x110/0x180 [165333.282186] [] __alloc_pages_slowpath+0x6b6/0x724 [165333.282187] [] __alloc_pages_nodemask+0x405/0x420 [165333.282189] [] dma_generic_alloc_coherent+0x8f/0x140 [165333.282191] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165333.282199] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165333.282204] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165333.282209] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165333.282214] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165333.282218] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165333.282222] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165333.282228] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165333.282230] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165333.282234] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165333.282237] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165333.282240] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165333.282241] [] process_one_work+0x17a/0x440 [165333.282242] [] worker_thread+0x126/0x3c0 [165333.282244] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165333.282245] [] kthread+0xcf/0xe0 [165333.282246] [] ? insert_kthread_work+0x40/0x40 [165333.282247] [] ret_from_fork+0x58/0x90 [165333.282248] [] ? insert_kthread_work+0x40/0x40 [165333.282249] Mem-Info: [165333.282253] active_anon:66 inactive_anon:0 isolated_anon:0 active_file:83 inactive_file:103 isolated_file:0 unevictable:6774 dirty:0 writeback:100 unstable:0 slab_reclaimable:16019 slab_unreclaimable:564312 mapped:1622 shmem:1 pagetables:1559 bounce:0 free:39376 free_pcp:9 free_cma:0 [165333.282257] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165333.282258] lowmem_reserve[]: 0 2580 15620 15620 [165333.282262] Node 0 DMA32 free:59540kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:12kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:16kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:8 all_unreclaimable? no [165333.282263] lowmem_reserve[]: 0 0 13040 13040 [165333.282266] Node 0 Normal free:36624kB min:37012kB low:46264kB high:55516kB active_anon:16kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:80kB mapped:1504kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101696kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? no [165333.282267] lowmem_reserve[]: 0 0 0 0 [165333.282271] Node 1 Normal free:45760kB min:45728kB low:57160kB high:68592kB active_anon:248kB inactive_anon:240kB active_file:456kB inactive_file:484kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:304kB mapped:4644kB shmem:4kB slab_reclaimable:33272kB slab_unreclaimable:958692kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:32kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:218 all_unreclaimable? no [165333.282272] lowmem_reserve[]: 0 0 0 0 [165333.282277] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165333.282282] Node 0 DMA32: 88*4kB (UEM) 2345*8kB (UEM) 1295*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59576kB [165333.282286] Node 0 Normal: 254*4kB (UEM) 175*8kB (UEM) 213*16kB (UEM) 289*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37664kB [165333.282290] Node 1 Normal: 515*4kB (UEM) 369*8kB (EM) 215*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47588kB [165333.282291] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165333.282292] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165333.282292] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165333.282293] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165333.282293] 2080 total pagecache pages [165333.282294] 50 pages in swap cache [165333.282294] Swap cache stats: add 301361, delete 301311, find 89736/144440 [165333.282295] Free swap = 16247900kB [165333.282295] Total swap = 16319484kB [165333.282296] 8369063 pages RAM [165333.282296] 0 pages HighMem/MovableOnly [165333.282296] 241261 pages reserved [165333.282418] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165333.282420] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165333.282420] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165333.282423] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165333.282424] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165333.282425] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165333.282426] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165333.282426] Call Trace: [165333.282428] [] dump_stack+0x19/0x1b [165333.282429] [] warn_alloc_failed+0x110/0x180 [165333.282430] [] ? drain_pages+0xb0/0xb0 [165333.282432] [] __alloc_pages_slowpath+0x6b6/0x724 [165333.282433] [] __alloc_pages_nodemask+0x405/0x420 [165333.282434] [] alloc_pages_current+0x98/0x110 [165333.282436] [] __get_free_pages+0xe/0x40 [165333.282437] [] swiotlb_alloc_coherent+0x5e/0x150 [165333.282439] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165333.282445] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165333.282450] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165333.282454] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165333.282459] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165333.282462] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165333.282467] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165333.282472] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165333.282474] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165333.282477] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165333.282481] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165333.282483] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165333.282485] [] process_one_work+0x17a/0x440 [165333.282486] [] worker_thread+0x126/0x3c0 [165333.282487] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165333.282488] [] kthread+0xcf/0xe0 [165333.282490] [] ? insert_kthread_work+0x40/0x40 [165333.282491] [] ret_from_fork+0x58/0x90 [165333.282492] [] ? insert_kthread_work+0x40/0x40 [165333.282492] Mem-Info: [165333.282496] active_anon:66 inactive_anon:0 isolated_anon:0 active_file:83 inactive_file:103 isolated_file:0 unevictable:6774 dirty:0 writeback:100 unstable:0 slab_reclaimable:16019 slab_unreclaimable:564312 mapped:1622 shmem:1 pagetables:1559 bounce:0 free:39376 free_pcp:0 free_cma:0 [165333.282499] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165333.282500] lowmem_reserve[]: 0 2580 15620 15620 [165333.282505] Node 0 DMA32 free:59540kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:12kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:16kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2 all_unreclaimable? no [165333.282506] lowmem_reserve[]: 0 0 13040 13040 [165333.282510] Node 0 Normal free:36624kB min:37012kB low:46264kB high:55516kB active_anon:16kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:80kB mapped:1504kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101696kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:324 all_unreclaimable? yes [165333.282511] lowmem_reserve[]: 0 0 0 0 [165333.282514] Node 1 Normal free:45760kB min:45728kB low:57160kB high:68592kB active_anon:248kB inactive_anon:240kB active_file:456kB inactive_file:484kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:304kB mapped:4644kB shmem:4kB slab_reclaimable:33272kB slab_unreclaimable:958692kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:130 all_unreclaimable? no [165333.282515] lowmem_reserve[]: 0 0 0 0 [165333.282520] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165333.282525] Node 0 DMA32: 89*4kB (UEM) 2343*8kB (UE) 1295*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59596kB [165333.282529] Node 0 Normal: 254*4kB (UEM) 175*8kB (UEM) 213*16kB (UEM) 289*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37664kB [165333.282534] Node 1 Normal: 523*4kB (UEM) 369*8kB (EM) 215*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47620kB [165333.282534] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165333.282535] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165333.282536] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165333.282536] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165333.282537] 2080 total pagecache pages [165333.282537] 47 pages in swap cache [165333.282538] Swap cache stats: add 301361, delete 301314, find 89736/144440 [165333.282538] Free swap = 16247900kB [165333.282538] Total swap = 16319484kB [165333.282539] 8369063 pages RAM [165333.282539] 0 pages HighMem/MovableOnly [165333.282540] 241261 pages reserved [165333.336767] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165333.336768] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165333.336769] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165333.336776] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165333.336777] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165333.336778] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165333.336779] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165333.336780] Call Trace: [165333.336782] [] dump_stack+0x19/0x1b [165333.336784] [] warn_alloc_failed+0x110/0x180 [165333.336785] [] __alloc_pages_slowpath+0x6b6/0x724 [165333.336787] [] __alloc_pages_nodemask+0x405/0x420 [165333.336789] [] dma_generic_alloc_coherent+0x8f/0x140 [165333.336790] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165333.336798] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165333.336809] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165333.336815] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165333.336819] [] ? mlx4_ib_create_qp+0x350/0xdc0 [mlx4_ib] [165333.336823] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165333.336827] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165333.336832] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165333.336837] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165333.336839] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165333.336843] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165333.336847] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165333.336849] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165333.336851] [] process_one_work+0x17a/0x440 [165333.336852] [] worker_thread+0x126/0x3c0 [165333.336853] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165333.336854] [] kthread+0xcf/0xe0 [165333.336856] [] ? insert_kthread_work+0x40/0x40 [165333.336857] [] ret_from_fork+0x58/0x90 [165333.336858] [] ? insert_kthread_work+0x40/0x40 [165333.336858] Mem-Info: [165333.336862] active_anon:39 inactive_anon:0 isolated_anon:0 active_file:75 inactive_file:43 isolated_file:0 unevictable:6774 dirty:0 writeback:85 unstable:0 slab_reclaimable:16019 slab_unreclaimable:564312 mapped:1593 shmem:1 pagetables:1559 bounce:0 free:39864 free_pcp:0 free_cma:0 [165333.336866] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165333.336868] lowmem_reserve[]: 0 2580 15620 15620 [165333.336872] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:20kB inactive_anon:0kB active_file:0kB inactive_file:52kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:16kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165333.336873] lowmem_reserve[]: 0 0 13040 13040 [165333.336877] Node 0 Normal free:37224kB min:37012kB low:46264kB high:55516kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:76kB mapped:1508kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101696kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165333.336878] lowmem_reserve[]: 0 0 0 0 [165333.336882] Node 1 Normal free:47172kB min:45728kB low:57160kB high:68592kB active_anon:128kB inactive_anon:144kB active_file:400kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:248kB mapped:4524kB shmem:4kB slab_reclaimable:33272kB slab_unreclaimable:958692kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165333.336882] lowmem_reserve[]: 0 0 0 0 [165333.336888] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165333.336892] Node 0 DMA32: 87*4kB (UE) 2344*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59580kB [165333.336897] Node 0 Normal: 153*4kB (UEM) 175*8kB (UEM) 213*16kB (UEM) 289*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37260kB [165333.336901] Node 1 Normal: 564*4kB (UEM) 379*8kB (UEM) 216*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47880kB [165333.336902] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165333.336902] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165333.336903] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165333.336903] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165333.336904] 1936 total pagecache pages [165333.336904] 54 pages in swap cache [165333.336905] Swap cache stats: add 301416, delete 301362, find 89753/144465 [165333.336905] Free swap = 16247896kB [165333.336906] Total swap = 16319484kB [165333.336906] 8369063 pages RAM [165333.336907] 0 pages HighMem/MovableOnly [165333.336907] 241261 pages reserved [165333.337027] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165333.337028] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165333.337028] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165333.337031] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165333.337032] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165333.337033] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165333.337034] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165333.337034] Call Trace: [165333.337036] [] dump_stack+0x19/0x1b [165333.337038] [] warn_alloc_failed+0x110/0x180 [165333.337039] [] ? drain_pages+0xb0/0xb0 [165333.337040] [] __alloc_pages_slowpath+0x6b6/0x724 [165333.337041] [] __alloc_pages_nodemask+0x405/0x420 [165333.337043] [] alloc_pages_current+0x98/0x110 [165333.337044] [] __get_free_pages+0xe/0x40 [165333.337046] [] swiotlb_alloc_coherent+0x5e/0x150 [165333.337047] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165333.337054] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165333.337059] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165333.337063] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165333.337067] [] ? mlx4_ib_create_qp+0x350/0xdc0 [mlx4_ib] [165333.337071] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165333.337074] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165333.337079] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165333.337084] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165333.337086] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165333.337089] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165333.337093] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165333.337095] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165333.337097] [] process_one_work+0x17a/0x440 [165333.337098] [] worker_thread+0x126/0x3c0 [165333.337099] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165333.337100] [] kthread+0xcf/0xe0 [165333.337101] [] ? insert_kthread_work+0x40/0x40 [165333.337103] [] ret_from_fork+0x58/0x90 [165333.337104] [] ? insert_kthread_work+0x40/0x40 [165333.337104] Mem-Info: [165333.337108] active_anon:39 inactive_anon:0 isolated_anon:0 active_file:75 inactive_file:43 isolated_file:0 unevictable:6774 dirty:0 writeback:85 unstable:0 slab_reclaimable:16019 slab_unreclaimable:564312 mapped:1593 shmem:1 pagetables:1559 bounce:0 free:39864 free_pcp:0 free_cma:0 [165333.337111] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165333.337112] lowmem_reserve[]: 0 2580 15620 15620 [165333.337116] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:20kB inactive_anon:0kB active_file:0kB inactive_file:52kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:16kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165333.337117] lowmem_reserve[]: 0 0 13040 13040 [165333.337120] Node 0 Normal free:37224kB min:37012kB low:46264kB high:55516kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:76kB mapped:1508kB shmem:0kB slab_reclaimable:26520kB slab_unreclaimable:1101696kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165333.337121] lowmem_reserve[]: 0 0 0 0 [165333.337125] Node 1 Normal free:47172kB min:45728kB low:57160kB high:68592kB active_anon:128kB inactive_anon:144kB active_file:400kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:248kB mapped:4524kB shmem:4kB slab_reclaimable:33272kB slab_unreclaimable:958692kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165333.337126] lowmem_reserve[]: 0 0 0 0 [165333.337131] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165333.337135] Node 0 DMA32: 87*4kB (UE) 2344*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59580kB [165333.337140] Node 0 Normal: 153*4kB (UEM) 175*8kB (UEM) 213*16kB (UEM) 289*32kB (UEM) 313*64kB (UM) 18*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37260kB [165333.337144] Node 1 Normal: 564*4kB (UEM) 379*8kB (UEM) 216*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47880kB [165333.337145] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165333.337145] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165333.337146] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165333.337146] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165333.337147] 1936 total pagecache pages [165333.337147] 54 pages in swap cache [165333.337148] Swap cache stats: add 301416, delete 301362, find 89753/144465 [165333.337148] Free swap = 16247896kB [165333.337148] Total swap = 16319484kB [165333.337149] 8369063 pages RAM [165333.337149] 0 pages HighMem/MovableOnly [165333.337149] 241261 pages reserved [165335.368169] warn_alloc_failed: 2 callbacks suppressed [165335.368169] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165335.368171] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165335.368172] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165335.368181] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165335.368183] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165335.368184] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165335.368185] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165335.368185] Call Trace: [165335.368191] [] dump_stack+0x19/0x1b [165335.368195] [] warn_alloc_failed+0x110/0x180 [165335.368197] [] __alloc_pages_slowpath+0x6b6/0x724 [165335.368198] [] __alloc_pages_nodemask+0x405/0x420 [165335.368201] [] dma_generic_alloc_coherent+0x8f/0x140 [165335.368204] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165335.368223] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165335.368228] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165335.368234] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165335.368239] [] ? mlx4_ib_create_qp+0x350/0xdc0 [mlx4_ib] [165335.368243] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165335.368247] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165335.368251] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165335.368258] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165335.368261] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165335.368267] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165335.368272] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165335.368274] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165335.368277] [] process_one_work+0x17a/0x440 [165335.368278] [] worker_thread+0x126/0x3c0 [165335.368279] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165335.368281] [] kthread+0xcf/0xe0 [165335.368282] [] ? insert_kthread_work+0x40/0x40 [165335.368283] [] ret_from_fork+0x58/0x90 [165335.368284] [] ? insert_kthread_work+0x40/0x40 [165335.368285] Mem-Info: [165335.368289] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:23 inactive_file:48 isolated_file:0 unevictable:6774 dirty:0 writeback:109 unstable:0 slab_reclaimable:16035 slab_unreclaimable:564316 mapped:1550 shmem:0 pagetables:1559 bounce:0 free:39941 free_pcp:0 free_cma:0 [165335.368294] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165335.368295] lowmem_reserve[]: 0 2580 15620 15620 [165335.368299] Node 0 DMA32 free:59572kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:16kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:36kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165335.368300] lowmem_reserve[]: 0 0 13040 13040 [165335.368303] Node 0 Normal free:36656kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:116kB mapped:1508kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101696kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165335.368304] lowmem_reserve[]: 0 0 0 0 [165335.368308] Node 1 Normal free:47956kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:52kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:284kB mapped:4352kB shmem:0kB slab_reclaimable:33272kB slab_unreclaimable:958708kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165335.368308] lowmem_reserve[]: 0 0 0 0 [165335.368314] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165335.368318] Node 0 DMA32: 92*4kB (UEM) 2347*8kB (UEM) 1295*16kB (UEM) 459*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165335.368323] Node 0 Normal: 183*4kB (UEM) 170*8kB (UEM) 213*16kB (UEM) 289*32kB (UEM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37276kB [165335.368327] Node 1 Normal: 566*4kB (UEM) 379*8kB (UEM) 215*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47872kB [165335.368328] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165335.368328] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165335.368329] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165335.368329] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165335.368330] 1756 total pagecache pages [165335.368331] 55 pages in swap cache [165335.368331] Swap cache stats: add 302022, delete 301967, find 89929/144762 [165335.368332] Free swap = 16247896kB [165335.368332] Total swap = 16319484kB [165335.368333] 8369063 pages RAM [165335.368333] 0 pages HighMem/MovableOnly [165335.368333] 241261 pages reserved [165335.379220] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165335.379223] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165335.379224] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165335.379233] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165335.379235] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165335.379236] ffff880406203850 ffffffff81188820 0000000000000000 ffff88043ffdb000 [165335.379237] 0000000000000008 00000000000080d0 ffff880406203850 0000000039b5c568 [165335.379237] Call Trace: [165335.379242] [] dump_stack+0x19/0x1b [165335.379246] [] warn_alloc_failed+0x110/0x180 [165335.379247] [] __alloc_pages_slowpath+0x6b6/0x724 [165335.379249] [] __alloc_pages_nodemask+0x405/0x420 [165335.379251] [] alloc_pages_current+0x98/0x110 [165335.379253] [] __get_free_pages+0xe/0x40 [165335.379255] [] swiotlb_alloc_coherent+0x5e/0x150 [165335.379258] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165335.379277] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165335.379282] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165335.379287] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165335.379292] [] ? mlx4_ib_create_qp+0x350/0xdc0 [mlx4_ib] [165335.379295] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165335.379299] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165335.379303] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165335.379311] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165335.379313] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165335.379320] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165335.379325] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165335.379327] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165335.379329] [] process_one_work+0x17a/0x440 [165335.379330] [] worker_thread+0x126/0x3c0 [165335.379332] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165335.379333] [] kthread+0xcf/0xe0 [165335.379334] [] ? insert_kthread_work+0x40/0x40 [165335.379335] [] ret_from_fork+0x58/0x90 [165335.379336] [] ? insert_kthread_work+0x40/0x40 [165335.379337] Mem-Info: [165335.379345] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:23 inactive_file:48 isolated_file:0 unevictable:6774 dirty:0 writeback:109 unstable:0 slab_reclaimable:16035 slab_unreclaimable:564316 mapped:1550 shmem:0 pagetables:1559 bounce:0 free:39945 free_pcp:0 free_cma:0 [165335.379349] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165335.379350] lowmem_reserve[]: 0 2580 15620 15620 [165335.379354] Node 0 DMA32 free:59588kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:16kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:36kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165335.379355] lowmem_reserve[]: 0 0 13040 13040 [165335.379359] Node 0 Normal free:36656kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:116kB mapped:1508kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101696kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:21 all_unreclaimable? yes [165335.379360] lowmem_reserve[]: 0 0 0 0 [165335.379363] Node 1 Normal free:47956kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:52kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:284kB mapped:4352kB shmem:0kB slab_reclaimable:33272kB slab_unreclaimable:958708kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:250 all_unreclaimable? no [165335.379364] lowmem_reserve[]: 0 0 0 0 [165335.379370] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165335.379374] Node 0 DMA32: 92*4kB (UEM) 2347*8kB (UEM) 1296*16kB (UEM) 458*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59592kB [165335.379378] Node 0 Normal: 183*4kB (UEM) 170*8kB (UEM) 213*16kB (UEM) 289*32kB (UEM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37276kB [165335.379382] Node 1 Normal: 566*4kB (UEM) 379*8kB (UEM) 215*16kB (UEM) 449*32kB (UEM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47872kB [165335.379383] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165335.379384] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165335.379384] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165335.379385] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165335.379385] 1756 total pagecache pages [165335.379386] 55 pages in swap cache [165335.379387] Swap cache stats: add 302022, delete 301967, find 89929/144762 [165335.379387] Free swap = 16247896kB [165335.379387] Total swap = 16319484kB [165335.379388] 8369063 pages RAM [165335.379388] 0 pages HighMem/MovableOnly [165335.379388] 241261 pages reserved [165336.665294] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165336.665295] CPU: 6 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165336.665296] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165336.665301] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165336.665303] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165336.665303] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165336.665304] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165336.665305] Call Trace: [165336.665308] [] dump_stack+0x19/0x1b [165336.665310] [] warn_alloc_failed+0x110/0x180 [165336.665311] [] __alloc_pages_slowpath+0x6b6/0x724 [165336.665313] [] __alloc_pages_nodemask+0x405/0x420 [165336.665315] [] dma_generic_alloc_coherent+0x8f/0x140 [165336.665317] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165336.665328] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165336.665333] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165336.665338] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165336.665341] [] ? dma_pool_free+0xa7/0xd0 [165336.665346] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165336.665349] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165336.665354] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165336.665360] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165336.665362] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165336.665366] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165336.665370] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165336.665372] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165336.665374] [] process_one_work+0x17a/0x440 [165336.665375] [] worker_thread+0x126/0x3c0 [165336.665376] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165336.665378] [] kthread+0xcf/0xe0 [165336.665379] [] ? insert_kthread_work+0x40/0x40 [165336.665380] [] ret_from_fork+0x58/0x90 [165336.665381] [] ? insert_kthread_work+0x40/0x40 [165336.665382] Mem-Info: [165336.665386] active_anon:36 inactive_anon:0 isolated_anon:0 active_file:73 inactive_file:123 isolated_file:0 unevictable:6774 dirty:0 writeback:121 unstable:0 slab_reclaimable:16043 slab_unreclaimable:564316 mapped:1576 shmem:0 pagetables:1559 bounce:0 free:39493 free_pcp:0 free_cma:0 [165336.665390] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165336.665391] lowmem_reserve[]: 0 2580 15620 15620 [165336.665395] Node 0 DMA32 free:59608kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:36kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? no [165336.665396] lowmem_reserve[]: 0 0 13040 13040 [165336.665400] Node 0 Normal free:37016kB min:37012kB low:46264kB high:55516kB active_anon:12kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:124kB mapped:1508kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101696kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165336.665401] lowmem_reserve[]: 0 0 0 0 [165336.665404] Node 1 Normal free:45768kB min:45728kB low:57160kB high:68592kB active_anon:124kB inactive_anon:224kB active_file:392kB inactive_file:708kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:324kB mapped:4456kB shmem:0kB slab_reclaimable:33304kB slab_unreclaimable:958708kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165336.665405] lowmem_reserve[]: 0 0 0 0 [165336.665410] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165336.665415] Node 0 DMA32: 92*4kB (UEM) 2347*8kB (UEM) 1295*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165336.665420] Node 0 Normal: 145*4kB (UEM) 140*8kB (UEM) 205*16kB (UEM) 289*32kB (UEM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36756kB [165336.665424] Node 1 Normal: 472*4kB (UEM) 310*8kB (UEM) 216*16kB (UEM) 448*32kB (UM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46928kB [165336.665425] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165336.665426] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165336.665426] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165336.665427] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165336.665427] 1938 total pagecache pages [165336.665428] 131 pages in swap cache [165336.665428] Swap cache stats: add 302523, delete 302392, find 90077/145021 [165336.665429] Free swap = 16247984kB [165336.665429] Total swap = 16319484kB [165336.665430] 8369063 pages RAM [165336.665430] 0 pages HighMem/MovableOnly [165336.665430] 241261 pages reserved [165336.665501] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165336.665502] CPU: 6 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165336.665503] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165336.665505] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165336.665506] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165336.665507] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165336.665508] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165336.665508] Call Trace: [165336.665510] [] dump_stack+0x19/0x1b [165336.665512] [] warn_alloc_failed+0x110/0x180 [165336.665513] [] ? drain_pages+0xb0/0xb0 [165336.665514] [] __alloc_pages_slowpath+0x6b6/0x724 [165336.665516] [] __alloc_pages_nodemask+0x405/0x420 [165336.665517] [] alloc_pages_current+0x98/0x110 [165336.665519] [] __get_free_pages+0xe/0x40 [165336.665520] [] swiotlb_alloc_coherent+0x5e/0x150 [165336.665522] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165336.665528] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165336.665533] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165336.665538] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165336.665539] [] ? dma_pool_free+0xa7/0xd0 [165336.665543] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165336.665547] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165336.665551] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165336.665557] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165336.665559] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165336.665562] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165336.665566] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165336.665568] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165336.665570] [] process_one_work+0x17a/0x440 [165336.665571] [] worker_thread+0x126/0x3c0 [165336.665572] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165336.665573] [] kthread+0xcf/0xe0 [165336.665574] [] ? insert_kthread_work+0x40/0x40 [165336.665575] [] ret_from_fork+0x58/0x90 [165336.665576] [] ? insert_kthread_work+0x40/0x40 [165336.665577] Mem-Info: [165336.665580] active_anon:36 inactive_anon:0 isolated_anon:0 active_file:73 inactive_file:123 isolated_file:0 unevictable:6774 dirty:0 writeback:121 unstable:0 slab_reclaimable:16043 slab_unreclaimable:564316 mapped:1576 shmem:0 pagetables:1559 bounce:0 free:39493 free_pcp:0 free_cma:0 [165336.665584] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165336.665585] lowmem_reserve[]: 0 2580 15620 15620 [165336.665589] Node 0 DMA32 free:59608kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:36kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? no [165336.665590] lowmem_reserve[]: 0 0 13040 13040 [165336.665594] Node 0 Normal free:37016kB min:37012kB low:46264kB high:55516kB active_anon:12kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:124kB mapped:1508kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101696kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165336.665595] lowmem_reserve[]: 0 0 0 0 [165336.665599] Node 1 Normal free:45768kB min:45728kB low:57160kB high:68592kB active_anon:124kB inactive_anon:224kB active_file:392kB inactive_file:708kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:324kB mapped:4456kB shmem:0kB slab_reclaimable:33304kB slab_unreclaimable:958708kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165336.665599] lowmem_reserve[]: 0 0 0 0 [165336.665604] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165336.665609] Node 0 DMA32: 92*4kB (UEM) 2347*8kB (UEM) 1295*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165336.665613] Node 0 Normal: 145*4kB (UEM) 140*8kB (UEM) 205*16kB (UEM) 289*32kB (UEM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36756kB [165336.665617] Node 1 Normal: 472*4kB (UEM) 310*8kB (UEM) 216*16kB (UEM) 448*32kB (UM) 313*64kB (UEM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46928kB [165336.665618] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165336.665619] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165336.665619] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165336.665620] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165336.665620] 1938 total pagecache pages [165336.665620] 131 pages in swap cache [165336.665621] Swap cache stats: add 302523, delete 302392, find 90077/145021 [165336.665621] Free swap = 16247984kB [165336.665622] Total swap = 16319484kB [165336.665622] 8369063 pages RAM [165336.665622] 0 pages HighMem/MovableOnly [165336.665623] 241261 pages reserved [165337.665319] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165337.665321] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165337.665321] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165337.665325] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165337.665326] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165337.665327] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165337.665328] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165337.665329] Call Trace: [165337.665331] [] dump_stack+0x19/0x1b [165337.665333] [] warn_alloc_failed+0x110/0x180 [165337.665334] [] __alloc_pages_slowpath+0x6b6/0x724 [165337.665336] [] __alloc_pages_nodemask+0x405/0x420 [165337.665338] [] dma_generic_alloc_coherent+0x8f/0x140 [165337.665339] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165337.665347] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165337.665352] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165337.665357] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165337.665362] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165337.665366] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165337.665370] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165337.665376] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165337.665378] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165337.665382] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165337.665386] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165337.665388] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165337.665389] [] process_one_work+0x17a/0x440 [165337.665391] [] worker_thread+0x126/0x3c0 [165337.665392] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165337.665393] [] kthread+0xcf/0xe0 [165337.665394] [] ? insert_kthread_work+0x40/0x40 [165337.665395] [] ret_from_fork+0x58/0x90 [165337.665397] [] ? insert_kthread_work+0x40/0x40 [165337.665397] Mem-Info: [165337.665401] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:35 inactive_file:78 isolated_file:0 unevictable:6774 dirty:0 writeback:149 unstable:0 slab_reclaimable:16051 slab_unreclaimable:564324 mapped:1551 shmem:0 pagetables:1559 bounce:0 free:39475 free_pcp:0 free_cma:0 [165337.665405] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165337.665406] lowmem_reserve[]: 0 2580 15620 15620 [165337.665410] Node 0 DMA32 free:59612kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:36kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165337.665411] lowmem_reserve[]: 0 0 13040 13040 [165337.665414] Node 0 Normal free:36752kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:152kB mapped:1516kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101728kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165337.665415] lowmem_reserve[]: 0 0 0 0 [165337.665419] Node 1 Normal free:45956kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:408kB mapped:4348kB shmem:0kB slab_reclaimable:33336kB slab_unreclaimable:958708kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165337.665420] lowmem_reserve[]: 0 0 0 0 [165337.665425] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165337.665430] Node 0 DMA32: 91*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59612kB [165337.665434] Node 0 Normal: 246*4kB (UEM) 171*8kB (UEM) 204*16kB (UEM) 288*32kB (UEM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37360kB [165337.665438] Node 1 Normal: 479*4kB (UEM) 359*8kB (UEM) 210*16kB (UEM) 443*32kB (UEM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47028kB [165337.665439] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165337.665440] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165337.665440] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165337.665441] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165337.665441] 1855 total pagecache pages [165337.665442] 101 pages in swap cache [165337.665442] Swap cache stats: add 302648, delete 302547, find 90109/145075 [165337.665443] Free swap = 16247912kB [165337.665443] Total swap = 16319484kB [165337.665444] 8369063 pages RAM [165337.665444] 0 pages HighMem/MovableOnly [165337.665444] 241261 pages reserved [165337.665617] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165337.665619] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165337.665619] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165337.665626] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165337.665627] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165337.665628] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165337.665629] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165337.665629] Call Trace: [165337.665631] [] dump_stack+0x19/0x1b [165337.665633] [] warn_alloc_failed+0x110/0x180 [165337.665634] [] ? drain_pages+0xb0/0xb0 [165337.665635] [] __alloc_pages_slowpath+0x6b6/0x724 [165337.665636] [] __alloc_pages_nodemask+0x405/0x420 [165337.665638] [] alloc_pages_current+0x98/0x110 [165337.665639] [] __get_free_pages+0xe/0x40 [165337.665641] [] swiotlb_alloc_coherent+0x5e/0x150 [165337.665642] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165337.665649] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165337.665654] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165337.665658] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165337.665662] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165337.665666] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165337.665670] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165337.665676] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165337.665678] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165337.665681] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165337.665685] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165337.665687] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165337.665688] [] process_one_work+0x17a/0x440 [165337.665690] [] worker_thread+0x126/0x3c0 [165337.665691] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165337.665692] [] kthread+0xcf/0xe0 [165337.665693] [] ? insert_kthread_work+0x40/0x40 [165337.665694] [] ret_from_fork+0x58/0x90 [165337.665695] [] ? insert_kthread_work+0x40/0x40 [165337.665696] Mem-Info: [165337.665699] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:35 inactive_file:78 isolated_file:0 unevictable:6774 dirty:0 writeback:149 unstable:0 slab_reclaimable:16051 slab_unreclaimable:564324 mapped:1551 shmem:0 pagetables:1559 bounce:0 free:39475 free_pcp:0 free_cma:0 [165337.665703] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165337.665704] lowmem_reserve[]: 0 2580 15620 15620 [165337.665707] Node 0 DMA32 free:59612kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:36kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165337.665708] lowmem_reserve[]: 0 0 13040 13040 [165337.665712] Node 0 Normal free:36752kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:152kB mapped:1516kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101728kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165337.665713] lowmem_reserve[]: 0 0 0 0 [165337.665717] Node 1 Normal free:45956kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:408kB mapped:4348kB shmem:0kB slab_reclaimable:33336kB slab_unreclaimable:958708kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165337.665718] lowmem_reserve[]: 0 0 0 0 [165337.665723] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165337.665727] Node 0 DMA32: 91*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59612kB [165337.665731] Node 0 Normal: 246*4kB (UEM) 171*8kB (UEM) 204*16kB (UEM) 288*32kB (UEM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37360kB [165337.665735] Node 1 Normal: 479*4kB (UEM) 359*8kB (UEM) 210*16kB (UEM) 443*32kB (UEM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47028kB [165337.665736] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165337.665737] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165337.665737] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165337.665738] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165337.665738] 1855 total pagecache pages [165337.665738] 101 pages in swap cache [165337.665739] Swap cache stats: add 302648, delete 302547, find 90109/145075 [165337.665739] Free swap = 16247912kB [165337.665740] Total swap = 16319484kB [165337.665740] 8369063 pages RAM [165337.665741] 0 pages HighMem/MovableOnly [165337.665741] 241261 pages reserved [165338.369210] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165338.369211] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165338.369212] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165338.369221] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165338.369223] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165338.369224] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165338.369225] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165338.369225] Call Trace: [165338.369230] [] dump_stack+0x19/0x1b [165338.369233] [] warn_alloc_failed+0x110/0x180 [165338.369235] [] __alloc_pages_slowpath+0x6b6/0x724 [165338.369236] [] __alloc_pages_nodemask+0x405/0x420 [165338.369239] [] dma_generic_alloc_coherent+0x8f/0x140 [165338.369242] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165338.369261] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165338.369266] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165338.369272] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165338.369275] [] ? dma_pool_free+0xa7/0xd0 [165338.369280] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165338.369283] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165338.369288] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165338.369295] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165338.369297] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165338.369304] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165338.369309] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165338.369311] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165338.369313] [] process_one_work+0x17a/0x440 [165338.369315] [] worker_thread+0x126/0x3c0 [165338.369316] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165338.369317] [] kthread+0xcf/0xe0 [165338.369319] [] ? insert_kthread_work+0x40/0x40 [165338.369320] [] ret_from_fork+0x58/0x90 [165338.369321] [] ? insert_kthread_work+0x40/0x40 [165338.369322] Mem-Info: [165338.369326] active_anon:17 inactive_anon:0 isolated_anon:0 active_file:39 inactive_file:78 isolated_file:0 unevictable:6774 dirty:0 writeback:143 unstable:0 slab_reclaimable:16051 slab_unreclaimable:564328 mapped:1555 shmem:0 pagetables:1559 bounce:0 free:39682 free_pcp:0 free_cma:0 [165338.369331] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165338.369332] lowmem_reserve[]: 0 2580 15620 15620 [165338.369336] Node 0 DMA32 free:59616kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:8kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165338.369337] lowmem_reserve[]: 0 0 13040 13040 [165338.369341] Node 0 Normal free:36788kB min:37012kB low:46264kB high:55516kB active_anon:16kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:140kB mapped:1520kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101728kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165338.369342] lowmem_reserve[]: 0 0 0 0 [165338.369346] Node 1 Normal free:46744kB min:45728kB low:57160kB high:68592kB active_anon:48kB inactive_anon:244kB active_file:248kB inactive_file:368kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:392kB mapped:4360kB shmem:0kB slab_reclaimable:33336kB slab_unreclaimable:958724kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165338.369347] lowmem_reserve[]: 0 0 0 0 [165338.369352] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165338.369357] Node 0 DMA32: 92*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59616kB [165338.369361] Node 0 Normal: 247*4kB (UEM) 172*8kB (UEM) 204*16kB (UEM) 288*32kB (UEM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37372kB [165338.369365] Node 1 Normal: 545*4kB (UEM) 363*8kB (UEM) 211*16kB (UEM) 442*32kB (UEM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47308kB [165338.369366] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165338.369367] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165338.369368] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165338.369368] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165338.369368] 1855 total pagecache pages [165338.369369] 96 pages in swap cache [165338.369370] Swap cache stats: add 302710, delete 302614, find 90121/145100 [165338.369370] Free swap = 16247892kB [165338.369371] Total swap = 16319484kB [165338.369371] 8369063 pages RAM [165338.369372] 0 pages HighMem/MovableOnly [165338.369372] 241261 pages reserved [165338.369503] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165338.369504] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165338.369505] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165338.369507] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165338.369509] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165338.369510] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165338.369510] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165338.369511] Call Trace: [165338.369513] [] dump_stack+0x19/0x1b [165338.369514] [] warn_alloc_failed+0x110/0x180 [165338.369515] [] ? drain_pages+0xb0/0xb0 [165338.369516] [] __alloc_pages_slowpath+0x6b6/0x724 [165338.369518] [] __alloc_pages_nodemask+0x405/0x420 [165338.369520] [] alloc_pages_current+0x98/0x110 [165338.369521] [] __get_free_pages+0xe/0x40 [165338.369524] [] swiotlb_alloc_coherent+0x5e/0x150 [165338.369525] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165338.369532] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165338.369537] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165338.369541] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165338.369543] [] ? dma_pool_free+0xa7/0xd0 [165338.369547] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165338.369551] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165338.369555] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165338.369560] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165338.369563] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165338.369566] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165338.369569] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165338.369572] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165338.369573] [] process_one_work+0x17a/0x440 [165338.369574] [] worker_thread+0x126/0x3c0 [165338.369576] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165338.369577] [] kthread+0xcf/0xe0 [165338.369578] [] ? insert_kthread_work+0x40/0x40 [165338.369579] [] ret_from_fork+0x58/0x90 [165338.369580] [] ? insert_kthread_work+0x40/0x40 [165338.369580] Mem-Info: [165338.369584] active_anon:17 inactive_anon:0 isolated_anon:0 active_file:39 inactive_file:78 isolated_file:0 unevictable:6774 dirty:0 writeback:143 unstable:0 slab_reclaimable:16051 slab_unreclaimable:564328 mapped:1555 shmem:0 pagetables:1559 bounce:0 free:39682 free_pcp:0 free_cma:0 [165338.369587] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165338.369588] lowmem_reserve[]: 0 2580 15620 15620 [165338.369592] Node 0 DMA32 free:59616kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:8kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165338.369593] lowmem_reserve[]: 0 0 13040 13040 [165338.369597] Node 0 Normal free:36788kB min:37012kB low:46264kB high:55516kB active_anon:16kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:140kB mapped:1520kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101728kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165338.369603] lowmem_reserve[]: 0 0 0 0 [165338.369607] Node 1 Normal free:46744kB min:45728kB low:57160kB high:68592kB active_anon:48kB inactive_anon:244kB active_file:248kB inactive_file:368kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:392kB mapped:4360kB shmem:0kB slab_reclaimable:33336kB slab_unreclaimable:958724kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165338.369608] lowmem_reserve[]: 0 0 0 0 [165338.369613] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165338.369618] Node 0 DMA32: 92*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59616kB [165338.369622] Node 0 Normal: 247*4kB (UEM) 172*8kB (UEM) 204*16kB (UEM) 288*32kB (UEM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37372kB [165338.369626] Node 1 Normal: 545*4kB (UEM) 363*8kB (UEM) 211*16kB (UEM) 442*32kB (UEM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47308kB [165338.369627] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165338.369628] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165338.369628] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165338.369629] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165338.369629] 1855 total pagecache pages [165338.369630] 96 pages in swap cache [165338.369630] Swap cache stats: add 302710, delete 302614, find 90121/145100 [165338.369630] Free swap = 16247892kB [165338.369631] Total swap = 16319484kB [165338.369631] 8369063 pages RAM [165338.369632] 0 pages HighMem/MovableOnly [165338.369632] 241261 pages reserved [165339.667532] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165339.667533] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165339.667534] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165339.667538] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165339.667539] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165339.667540] ffff8804062038a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165339.667541] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165339.667541] Call Trace: [165339.667544] [] dump_stack+0x19/0x1b [165339.667546] [] warn_alloc_failed+0x110/0x180 [165339.667547] [] ? drain_pages+0xb0/0xb0 [165339.667548] [] __alloc_pages_slowpath+0x6b6/0x724 [165339.667550] [] __alloc_pages_nodemask+0x405/0x420 [165339.667552] [] dma_generic_alloc_coherent+0x8f/0x140 [165339.667557] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165339.667567] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165339.667572] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165339.667577] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165339.667582] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165339.667586] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165339.667590] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165339.667596] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165339.667598] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165339.667602] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165339.667605] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165339.667608] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165339.667609] [] process_one_work+0x17a/0x440 [165339.667610] [] worker_thread+0x126/0x3c0 [165339.667612] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165339.667613] [] kthread+0xcf/0xe0 [165339.667614] [] ? insert_kthread_work+0x40/0x40 [165339.667615] [] ret_from_fork+0x58/0x90 [165339.667616] [] ? insert_kthread_work+0x40/0x40 [165339.667617] Mem-Info: [165339.667621] active_anon:45 inactive_anon:7 isolated_anon:0 active_file:67 inactive_file:87 isolated_file:0 unevictable:6774 dirty:0 writeback:159 unstable:0 slab_reclaimable:16051 slab_unreclaimable:564328 mapped:1588 shmem:0 pagetables:1559 bounce:0 free:39460 free_pcp:0 free_cma:0 [165339.667624] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165339.667626] lowmem_reserve[]: 0 2580 15620 15620 [165339.667629] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:16kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165339.667630] lowmem_reserve[]: 0 0 13040 13040 [165339.667634] Node 0 Normal free:37100kB min:37012kB low:46264kB high:55516kB active_anon:16kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:140kB mapped:1508kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101728kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165339.667635] lowmem_reserve[]: 0 0 0 0 [165339.667638] Node 1 Normal free:45680kB min:45728kB low:57160kB high:68592kB active_anon:156kB inactive_anon:280kB active_file:364kB inactive_file:572kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:456kB mapped:4504kB shmem:0kB slab_reclaimable:33336kB slab_unreclaimable:958724kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:41 all_unreclaimable? no [165339.667639] lowmem_reserve[]: 0 0 0 0 [165339.667645] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165339.667649] Node 0 DMA32: 58*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59480kB [165339.667654] Node 0 Normal: 177*4kB (UEM) 164*8kB (UEM) 209*16kB (UEM) 287*32kB (UM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37076kB [165339.667658] Node 1 Normal: 540*4kB (UEM) 348*8kB (UEM) 217*16kB (UEM) 448*32kB (UEM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47456kB [165339.667659] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165339.667659] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165339.667660] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165339.667660] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165339.667661] 1942 total pagecache pages [165339.667661] 102 pages in swap cache [165339.667662] Swap cache stats: add 303067, delete 302965, find 90240/145310 [165339.667662] Free swap = 16247936kB [165339.667663] Total swap = 16319484kB [165339.667663] 8369063 pages RAM [165339.667663] 0 pages HighMem/MovableOnly [165339.667664] 241261 pages reserved [165339.667789] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165339.667790] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165339.667791] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165339.667793] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165339.667795] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165339.667796] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165339.667797] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165339.667797] Call Trace: [165339.667799] [] dump_stack+0x19/0x1b [165339.667801] [] warn_alloc_failed+0x110/0x180 [165339.667802] [] ? drain_pages+0xb0/0xb0 [165339.667803] [] __alloc_pages_slowpath+0x6b6/0x724 [165339.667804] [] __alloc_pages_nodemask+0x405/0x420 [165339.667806] [] alloc_pages_current+0x98/0x110 [165339.667807] [] __get_free_pages+0xe/0x40 [165339.667809] [] swiotlb_alloc_coherent+0x5e/0x150 [165339.667810] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165339.667816] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165339.667821] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165339.667826] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165339.667830] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165339.667834] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165339.667838] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165339.667843] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165339.667845] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165339.667849] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165339.667852] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165339.667854] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165339.667856] [] process_one_work+0x17a/0x440 [165339.667857] [] worker_thread+0x126/0x3c0 [165339.667858] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165339.667859] [] kthread+0xcf/0xe0 [165339.667861] [] ? insert_kthread_work+0x40/0x40 [165339.667862] [] ret_from_fork+0x58/0x90 [165339.667863] [] ? insert_kthread_work+0x40/0x40 [165339.667863] Mem-Info: [165339.667867] active_anon:45 inactive_anon:7 isolated_anon:0 active_file:67 inactive_file:87 isolated_file:0 unevictable:6774 dirty:0 writeback:159 unstable:0 slab_reclaimable:16051 slab_unreclaimable:564328 mapped:1588 shmem:0 pagetables:1559 bounce:0 free:39460 free_pcp:0 free_cma:0 [165339.667870] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165339.667872] lowmem_reserve[]: 0 2580 15620 15620 [165339.667875] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:16kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165339.667877] lowmem_reserve[]: 0 0 13040 13040 [165339.667880] Node 0 Normal free:37100kB min:37012kB low:46264kB high:55516kB active_anon:16kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:140kB mapped:1508kB shmem:0kB slab_reclaimable:26584kB slab_unreclaimable:1101728kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165339.667881] lowmem_reserve[]: 0 0 0 0 [165339.667885] Node 1 Normal free:45680kB min:45728kB low:57160kB high:68592kB active_anon:156kB inactive_anon:280kB active_file:364kB inactive_file:572kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:456kB mapped:4504kB shmem:0kB slab_reclaimable:33336kB slab_unreclaimable:958724kB kernel_stack:17376kB pagetables:3732kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:41 all_unreclaimable? no [165339.667886] lowmem_reserve[]: 0 0 0 0 [165339.667891] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165339.667895] Node 0 DMA32: 58*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59480kB [165339.667899] Node 0 Normal: 177*4kB (UEM) 164*8kB (UEM) 209*16kB (UEM) 287*32kB (UM) 314*64kB (UEM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37076kB [165339.667903] Node 1 Normal: 540*4kB (UEM) 348*8kB (UEM) 217*16kB (UEM) 448*32kB (UEM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47456kB [165339.667904] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165339.667904] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165339.667905] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165339.667905] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165339.667906] 1942 total pagecache pages [165339.667906] 102 pages in swap cache [165339.667907] Swap cache stats: add 303067, delete 302965, find 90240/145310 [165339.667907] Free swap = 16247936kB [165339.667907] Total swap = 16319484kB [165339.667908] 8369063 pages RAM [165339.667908] 0 pages HighMem/MovableOnly [165339.667908] 241261 pages reserved [165346.365408] warn_alloc_failed: 2 callbacks suppressed [165346.365409] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165346.365411] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165346.365412] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165346.365422] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165346.365423] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165346.365424] ffff8807e687b8a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165346.365426] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165346.365426] Call Trace: [165346.365432] [] dump_stack+0x19/0x1b [165346.365436] [] warn_alloc_failed+0x110/0x180 [165346.365438] [] ? drain_pages+0xb0/0xb0 [165346.365439] [] __alloc_pages_slowpath+0x6b6/0x724 [165346.365441] [] __alloc_pages_nodemask+0x405/0x420 [165346.365444] [] dma_generic_alloc_coherent+0x8f/0x140 [165346.365447] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165346.365467] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165346.365472] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165346.365479] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165346.365483] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165346.365487] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165346.365492] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165346.365499] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165346.365502] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165346.365508] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165346.365513] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165346.365515] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165346.365518] [] process_one_work+0x17a/0x440 [165346.365519] [] worker_thread+0x126/0x3c0 [165346.365520] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165346.365522] [] kthread+0xcf/0xe0 [165346.365523] [] ? insert_kthread_work+0x40/0x40 [165346.365525] [] ret_from_fork+0x58/0x90 [165346.365526] [] ? insert_kthread_work+0x40/0x40 [165346.365526] Mem-Info: [165346.365531] active_anon:4 inactive_anon:7 isolated_anon:0 active_file:37 inactive_file:59 isolated_file:0 unevictable:6774 dirty:0 writeback:82 unstable:0 slab_reclaimable:16075 slab_unreclaimable:564336 mapped:1554 shmem:1 pagetables:1560 bounce:0 free:39840 free_pcp:58 free_cma:0 [165346.365535] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165346.365536] lowmem_reserve[]: 0 2580 15620 15620 [165346.365540] Node 0 DMA32 free:59584kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:16kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:48kB mapped:352kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:708 all_unreclaimable? yes [165346.365541] lowmem_reserve[]: 0 0 13040 13040 [165346.365544] Node 0 Normal free:37012kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:152kB mapped:1520kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101760kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165346.365545] lowmem_reserve[]: 0 0 0 0 [165346.365549] Node 1 Normal free:47184kB min:45728kB low:57160kB high:68592kB active_anon:16kB inactive_anon:268kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:128kB mapped:4344kB shmem:4kB slab_reclaimable:33368kB slab_unreclaimable:958724kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:232kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165346.365550] lowmem_reserve[]: 0 0 0 0 [165346.365555] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165346.365560] Node 0 DMA32: 90*4kB (UEM) 2347*8kB (UEM) 1294*16kB (UEM) 459*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59584kB [165346.365564] Node 0 Normal: 252*4kB (UEM) 164*8kB (UEM) 210*16kB (UEM) 287*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37328kB [165346.365568] Node 1 Normal: 415*4kB (EM) 351*8kB (EM) 218*16kB (UEM) 446*32kB (UM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46932kB [165346.365569] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165346.365570] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165346.365571] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165346.365571] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165346.365571] 1839 total pagecache pages [165346.365572] 131 pages in swap cache [165346.365573] Swap cache stats: add 303844, delete 303713, find 90462/145696 [165346.365573] Free swap = 16247876kB [165346.365574] Total swap = 16319484kB [165346.365574] 8369063 pages RAM [165346.365575] 0 pages HighMem/MovableOnly [165346.365575] 241261 pages reserved [165346.365710] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165346.365712] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165346.365712] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165346.365714] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165346.365716] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165346.365717] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165346.365718] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165346.365718] Call Trace: [165346.365720] [] dump_stack+0x19/0x1b [165346.365721] [] warn_alloc_failed+0x110/0x180 [165346.365722] [] ? drain_pages+0xb0/0xb0 [165346.365723] [] __alloc_pages_slowpath+0x6b6/0x724 [165346.365725] [] __alloc_pages_nodemask+0x405/0x420 [165346.365727] [] alloc_pages_current+0x98/0x110 [165346.365728] [] __get_free_pages+0xe/0x40 [165346.365731] [] swiotlb_alloc_coherent+0x5e/0x150 [165346.365733] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165346.365739] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165346.365744] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165346.365748] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165346.365752] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165346.365756] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165346.365760] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165346.365766] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165346.365768] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165346.365771] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165346.365775] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165346.365777] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165346.365778] [] process_one_work+0x17a/0x440 [165346.365779] [] worker_thread+0x126/0x3c0 [165346.365781] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165346.365782] [] kthread+0xcf/0xe0 [165346.365783] [] ? insert_kthread_work+0x40/0x40 [165346.365784] [] ret_from_fork+0x58/0x90 [165346.365785] [] ? insert_kthread_work+0x40/0x40 [165346.365785] Mem-Info: [165346.365789] active_anon:4 inactive_anon:7 isolated_anon:0 active_file:37 inactive_file:59 isolated_file:0 unevictable:6774 dirty:0 writeback:82 unstable:0 slab_reclaimable:16075 slab_unreclaimable:564336 mapped:1554 shmem:1 pagetables:1560 bounce:0 free:39840 free_pcp:0 free_cma:0 [165346.365792] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165346.365793] lowmem_reserve[]: 0 2580 15620 15620 [165346.365797] Node 0 DMA32 free:59584kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:16kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:48kB mapped:352kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:708 all_unreclaimable? yes [165346.365798] lowmem_reserve[]: 0 0 13040 13040 [165346.365801] Node 0 Normal free:37012kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:152kB mapped:1520kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101760kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165346.365802] lowmem_reserve[]: 0 0 0 0 [165346.365805] Node 1 Normal free:47184kB min:45728kB low:57160kB high:68592kB active_anon:16kB inactive_anon:268kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:128kB mapped:4344kB shmem:4kB slab_reclaimable:33368kB slab_unreclaimable:958724kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165346.365806] lowmem_reserve[]: 0 0 0 0 [165346.365811] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165346.365815] Node 0 DMA32: 90*4kB (UEM) 2347*8kB (UEM) 1294*16kB (UEM) 459*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59584kB [165346.365820] Node 0 Normal: 252*4kB (UEM) 164*8kB (UEM) 210*16kB (UEM) 287*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37328kB [165346.365824] Node 1 Normal: 473*4kB (EM) 351*8kB (EM) 218*16kB (UEM) 446*32kB (UM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47164kB [165346.365825] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165346.365825] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165346.365826] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165346.365826] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165346.365827] 1839 total pagecache pages [165346.365827] 131 pages in swap cache [165346.365827] Swap cache stats: add 303844, delete 303713, find 90462/145696 [165346.365828] Free swap = 16247876kB [165346.365828] Total swap = 16319484kB [165346.365829] 8369063 pages RAM [165346.365829] 0 pages HighMem/MovableOnly [165346.365829] 241261 pages reserved [165346.378178] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165346.378179] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165346.378180] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165346.378184] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165346.378186] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165346.378187] ffff8807e687b8a0 ffffffff81188820 0000000000000000 ffff88043ffdb000 [165346.378188] 0000000000000008 00000000000080d0 ffff8807e687b8a0 00000000c340495d [165346.378188] Call Trace: [165346.378191] [] dump_stack+0x19/0x1b [165346.378193] [] warn_alloc_failed+0x110/0x180 [165346.378194] [] __alloc_pages_slowpath+0x6b6/0x724 [165346.378196] [] __alloc_pages_nodemask+0x405/0x420 [165346.378198] [] dma_generic_alloc_coherent+0x8f/0x140 [165346.378199] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165346.378208] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165346.378213] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165346.378218] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165346.378222] [] ? dma_pool_free+0xa7/0xd0 [165346.378226] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165346.378230] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165346.378234] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165346.378240] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165346.378243] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165346.378247] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165346.378251] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165346.378253] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165346.378254] [] process_one_work+0x17a/0x440 [165346.378256] [] worker_thread+0x126/0x3c0 [165346.378257] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165346.378258] [] kthread+0xcf/0xe0 [165346.378259] [] ? insert_kthread_work+0x40/0x40 [165346.378260] [] ret_from_fork+0x58/0x90 [165346.378261] [] ? insert_kthread_work+0x40/0x40 [165346.378262] Mem-Info: [165346.378266] active_anon:4 inactive_anon:7 isolated_anon:0 active_file:37 inactive_file:59 isolated_file:0 unevictable:6774 dirty:0 writeback:82 unstable:0 slab_reclaimable:16075 slab_unreclaimable:564336 mapped:1554 shmem:1 pagetables:1560 bounce:0 free:39807 free_pcp:0 free_cma:0 [165346.378269] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165346.378271] lowmem_reserve[]: 0 2580 15620 15620 [165346.378274] Node 0 DMA32 free:59536kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:16kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:48kB mapped:352kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165346.378275] lowmem_reserve[]: 0 0 13040 13040 [165346.378279] Node 0 Normal free:37012kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:152kB mapped:1520kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101760kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165346.378280] lowmem_reserve[]: 0 0 0 0 [165346.378283] Node 1 Normal free:47100kB min:45728kB low:57160kB high:68592kB active_anon:16kB inactive_anon:268kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:128kB mapped:4344kB shmem:4kB slab_reclaimable:33368kB slab_unreclaimable:958724kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:53 all_unreclaimable? no [165346.378284] lowmem_reserve[]: 0 0 0 0 [165346.378289] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165346.378294] Node 0 DMA32: 93*4kB (UEM) 2347*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165346.378298] Node 0 Normal: 251*4kB (UEM) 164*8kB (UEM) 210*16kB (UEM) 287*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37324kB [165346.378303] Node 1 Normal: 474*4kB (UEM) 353*8kB (UEM) 218*16kB (UEM) 445*32kB (UM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47152kB [165346.378304] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165346.378304] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165346.378305] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165346.378305] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165346.378306] 1839 total pagecache pages [165346.378306] 131 pages in swap cache [165346.378307] Swap cache stats: add 303844, delete 303713, find 90462/145696 [165346.378307] Free swap = 16247876kB [165346.378308] Total swap = 16319484kB [165346.378308] 8369063 pages RAM [165346.378308] 0 pages HighMem/MovableOnly [165346.378309] 241261 pages reserved [165346.378429] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165346.378430] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165346.378431] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165346.378434] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165346.378435] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165346.378436] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165346.378437] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165346.378437] Call Trace: [165346.378439] [] dump_stack+0x19/0x1b [165346.378441] [] warn_alloc_failed+0x110/0x180 [165346.378442] [] ? drain_pages+0xb0/0xb0 [165346.378443] [] __alloc_pages_slowpath+0x6b6/0x724 [165346.378444] [] __alloc_pages_nodemask+0x405/0x420 [165346.378445] [] alloc_pages_current+0x98/0x110 [165346.378447] [] __get_free_pages+0xe/0x40 [165346.378449] [] swiotlb_alloc_coherent+0x5e/0x150 [165346.378450] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165346.378457] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165346.378462] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165346.378466] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165346.378467] [] ? dma_pool_free+0xa7/0xd0 [165346.378471] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165346.378475] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165346.378479] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165346.378485] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165346.378487] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165346.378490] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165346.378494] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165346.378496] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165346.378497] [] process_one_work+0x17a/0x440 [165346.378498] [] worker_thread+0x126/0x3c0 [165346.378500] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165346.378501] [] kthread+0xcf/0xe0 [165346.378502] [] ? insert_kthread_work+0x40/0x40 [165346.378503] [] ret_from_fork+0x58/0x90 [165346.378504] [] ? insert_kthread_work+0x40/0x40 [165346.378504] Mem-Info: [165346.378508] active_anon:4 inactive_anon:7 isolated_anon:0 active_file:37 inactive_file:59 isolated_file:0 unevictable:6774 dirty:0 writeback:82 unstable:0 slab_reclaimable:16075 slab_unreclaimable:564336 mapped:1554 shmem:1 pagetables:1560 bounce:0 free:39807 free_pcp:0 free_cma:0 [165346.378511] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165346.378513] lowmem_reserve[]: 0 2580 15620 15620 [165346.378517] Node 0 DMA32 free:59536kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:16kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:48kB mapped:352kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165346.378518] lowmem_reserve[]: 0 0 13040 13040 [165346.378521] Node 0 Normal free:37012kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:152kB mapped:1520kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101760kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165346.378522] lowmem_reserve[]: 0 0 0 0 [165346.378526] Node 1 Normal free:47100kB min:45728kB low:57160kB high:68592kB active_anon:16kB inactive_anon:268kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:128kB mapped:4344kB shmem:4kB slab_reclaimable:33368kB slab_unreclaimable:958724kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:110 all_unreclaimable? no [165346.378526] lowmem_reserve[]: 0 0 0 0 [165346.378531] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165346.378536] Node 0 DMA32: 93*4kB (UEM) 2347*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165346.378540] Node 0 Normal: 251*4kB (UEM) 164*8kB (UEM) 210*16kB (UEM) 287*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37324kB [165346.378544] Node 1 Normal: 474*4kB (UEM) 353*8kB (UEM) 218*16kB (UEM) 445*32kB (UM) 312*64kB (UM) 37*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47152kB [165346.378545] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165346.378545] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165346.378546] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165346.378546] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165346.378546] 1839 total pagecache pages [165346.378547] 131 pages in swap cache [165346.378547] Swap cache stats: add 303844, delete 303713, find 90462/145696 [165346.378548] Free swap = 16247876kB [165346.378548] Total swap = 16319484kB [165346.378549] 8369063 pages RAM [165346.378549] 0 pages HighMem/MovableOnly [165346.378549] 241261 pages reserved [165348.290931] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165348.290933] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165348.290934] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165348.290938] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165348.290940] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165348.290941] ffff8807e15678a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165348.290942] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165348.290942] Call Trace: [165348.290945] [] dump_stack+0x19/0x1b [165348.290947] [] warn_alloc_failed+0x110/0x180 [165348.290948] [] ? drain_pages+0xb0/0xb0 [165348.290950] [] __alloc_pages_slowpath+0x6b6/0x724 [165348.290951] [] __alloc_pages_nodemask+0x405/0x420 [165348.290953] [] dma_generic_alloc_coherent+0x8f/0x140 [165348.290955] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165348.290963] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165348.290968] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165348.290974] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165348.290978] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165348.290982] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165348.290987] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165348.290992] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165348.290995] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165348.290999] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165348.291003] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165348.291005] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165348.291007] [] process_one_work+0x17a/0x440 [165348.291008] [] worker_thread+0x126/0x3c0 [165348.291009] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165348.291010] [] kthread+0xcf/0xe0 [165348.291012] [] ? insert_kthread_work+0x40/0x40 [165348.291013] [] ret_from_fork+0x58/0x90 [165348.291014] [] ? insert_kthread_work+0x40/0x40 [165348.291014] Mem-Info: [165348.291018] active_anon:23 inactive_anon:8 isolated_anon:0 active_file:100 inactive_file:4 isolated_file:0 unevictable:6774 dirty:0 writeback:92 unstable:0 slab_reclaimable:16083 slab_unreclaimable:564340 mapped:1553 shmem:0 pagetables:1560 bounce:0 free:39369 free_pcp:0 free_cma:0 [165348.291022] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165348.291023] lowmem_reserve[]: 0 2580 15620 15620 [165348.291027] Node 0 DMA32 free:59376kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:256kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:48kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1755 all_unreclaimable? no [165348.291028] lowmem_reserve[]: 0 0 13040 13040 [165348.291031] Node 0 Normal free:36508kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:144kB mapped:1520kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101760kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165348.291032] lowmem_reserve[]: 0 0 0 0 [165348.291036] Node 1 Normal free:46012kB min:45728kB low:57160kB high:68592kB active_anon:88kB inactive_anon:276kB active_file:296kB inactive_file:348kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:176kB mapped:4348kB shmem:0kB slab_reclaimable:33400kB slab_unreclaimable:958740kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:25 all_unreclaimable? no [165348.291037] lowmem_reserve[]: 0 0 0 0 [165348.291042] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165348.291047] Node 0 DMA32: 88*4kB (UEM) 2348*8kB (UEM) 1293*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59568kB [165348.291051] Node 0 Normal: 247*4kB (UEM) 176*8kB (UEM) 212*16kB (UEM) 287*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37436kB [165348.291055] Node 1 Normal: 514*4kB (EM) 368*8kB (EM) 217*16kB (UEM) 447*32kB (UEM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47416kB [165348.291056] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165348.291057] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165348.291058] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165348.291058] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165348.291059] 1824 total pagecache pages [165348.291059] 27 pages in swap cache [165348.291060] Swap cache stats: add 304375, delete 304348, find 90656/146012 [165348.291060] Free swap = 16247872kB [165348.291061] Total swap = 16319484kB [165348.291061] 8369063 pages RAM [165348.291061] 0 pages HighMem/MovableOnly [165348.291062] 241261 pages reserved [165348.291229] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165348.291231] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165348.291231] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165348.291233] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165348.291235] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165348.291236] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165348.291237] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165348.291237] Call Trace: [165348.291238] [] dump_stack+0x19/0x1b [165348.291240] [] warn_alloc_failed+0x110/0x180 [165348.291241] [] ? drain_pages+0xb0/0xb0 [165348.291242] [] __alloc_pages_slowpath+0x6b6/0x724 [165348.291244] [] __alloc_pages_nodemask+0x405/0x420 [165348.291245] [] alloc_pages_current+0x98/0x110 [165348.291247] [] __get_free_pages+0xe/0x40 [165348.291248] [] swiotlb_alloc_coherent+0x5e/0x150 [165348.291249] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165348.291261] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165348.291268] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165348.291273] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165348.291277] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165348.291281] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165348.291285] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165348.291290] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165348.291292] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165348.291296] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165348.291299] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165348.291302] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165348.291303] [] process_one_work+0x17a/0x440 [165348.291304] [] worker_thread+0x126/0x3c0 [165348.291306] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165348.291307] [] kthread+0xcf/0xe0 [165348.291308] [] ? insert_kthread_work+0x40/0x40 [165348.291309] [] ret_from_fork+0x58/0x90 [165348.291310] [] ? insert_kthread_work+0x40/0x40 [165348.291311] Mem-Info: [165348.291314] active_anon:23 inactive_anon:8 isolated_anon:0 active_file:77 inactive_file:27 isolated_file:0 unevictable:6774 dirty:0 writeback:92 unstable:0 slab_reclaimable:16083 slab_unreclaimable:564340 mapped:1553 shmem:0 pagetables:1560 bounce:0 free:39369 free_pcp:0 free_cma:0 [165348.291318] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165348.291319] lowmem_reserve[]: 0 2580 15620 15620 [165348.291322] Node 0 DMA32 free:59376kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:164kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:48kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1774 all_unreclaimable? no [165348.291323] lowmem_reserve[]: 0 0 13040 13040 [165348.291327] Node 0 Normal free:36508kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:144kB mapped:1520kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101760kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165348.291328] lowmem_reserve[]: 0 0 0 0 [165348.291332] Node 1 Normal free:46012kB min:45728kB low:57160kB high:68592kB active_anon:88kB inactive_anon:276kB active_file:296kB inactive_file:348kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:176kB mapped:4348kB shmem:0kB slab_reclaimable:33400kB slab_unreclaimable:958740kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no [165348.291333] lowmem_reserve[]: 0 0 0 0 [165348.291338] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165348.291343] Node 0 DMA32: 88*4kB (UEM) 2348*8kB (UEM) 1293*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59568kB [165348.291347] Node 0 Normal: 247*4kB (UEM) 176*8kB (UEM) 212*16kB (UEM) 287*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37436kB [165348.291351] Node 1 Normal: 544*4kB (EM) 368*8kB (EM) 217*16kB (UEM) 447*32kB (UEM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47536kB [165348.291352] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165348.291353] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165348.291353] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165348.291354] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165348.291354] 1824 total pagecache pages [165348.291355] 27 pages in swap cache [165348.291355] Swap cache stats: add 304375, delete 304348, find 90656/146012 [165348.291356] Free swap = 16247872kB [165348.291356] Total swap = 16319484kB [165348.291357] 8369063 pages RAM [165348.291357] 0 pages HighMem/MovableOnly [165348.291357] 241261 pages reserved [165349.329992] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165349.329994] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165349.329994] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165349.329998] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165349.329999] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165349.330000] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165349.330001] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165349.330002] Call Trace: [165349.330004] [] dump_stack+0x19/0x1b [165349.330006] [] warn_alloc_failed+0x110/0x180 [165349.330007] [] __alloc_pages_slowpath+0x6b6/0x724 [165349.330009] [] __alloc_pages_nodemask+0x405/0x420 [165349.330011] [] dma_generic_alloc_coherent+0x8f/0x140 [165349.330012] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165349.330020] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165349.330025] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165349.330031] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165349.330032] [] ? dma_pool_free+0xa7/0xd0 [165349.330037] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165349.330041] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165349.330045] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165349.330052] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165349.330054] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165349.330058] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165349.330062] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165349.330064] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165349.330065] [] process_one_work+0x17a/0x440 [165349.330066] [] worker_thread+0x126/0x3c0 [165349.330068] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165349.330069] [] kthread+0xcf/0xe0 [165349.330070] [] ? insert_kthread_work+0x40/0x40 [165349.330071] [] ret_from_fork+0x58/0x90 [165349.330072] [] ? insert_kthread_work+0x40/0x40 [165349.330073] Mem-Info: [165349.330077] active_anon:6 inactive_anon:0 isolated_anon:0 active_file:57 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:35 unstable:0 slab_reclaimable:16083 slab_unreclaimable:564340 mapped:1549 shmem:0 pagetables:1560 bounce:0 free:39838 free_pcp:0 free_cma:0 [165349.330080] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165349.330081] lowmem_reserve[]: 0 2580 15620 15620 [165349.330085] Node 0 DMA32 free:59612kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165349.330086] lowmem_reserve[]: 0 0 13040 13040 [165349.330090] Node 0 Normal free:36840kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:156kB mapped:1508kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101760kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165349.330091] lowmem_reserve[]: 0 0 0 0 [165349.330095] Node 1 Normal free:47320kB min:45728kB low:57160kB high:68592kB active_anon:24kB inactive_anon:128kB active_file:348kB inactive_file:128kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4348kB shmem:0kB slab_reclaimable:33400kB slab_unreclaimable:958740kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165349.330096] lowmem_reserve[]: 0 0 0 0 [165349.330101] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165349.330105] Node 0 DMA32: 90*4kB (UEM) 2346*8kB (UEM) 1293*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59624kB [165349.330110] Node 0 Normal: 184*4kB (UEM) 176*8kB (UEM) 212*16kB (UEM) 286*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37152kB [165349.330114] Node 1 Normal: 521*4kB (UEM) 379*8kB (UEM) 217*16kB (UEM) 447*32kB (UEM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47532kB [165349.330114] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165349.330115] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165349.330116] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165349.330116] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165349.330117] 1845 total pagecache pages [165349.330117] 70 pages in swap cache [165349.330118] Swap cache stats: add 304510, delete 304440, find 90689/146067 [165349.330118] Free swap = 16247876kB [165349.330118] Total swap = 16319484kB [165349.330119] 8369063 pages RAM [165349.330119] 0 pages HighMem/MovableOnly [165349.330120] 241261 pages reserved [165349.330261] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165349.330262] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165349.330262] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165349.330265] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165349.330266] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165349.330267] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165349.330268] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165349.330269] Call Trace: [165349.330270] [] dump_stack+0x19/0x1b [165349.330272] [] warn_alloc_failed+0x110/0x180 [165349.330273] [] ? drain_pages+0xb0/0xb0 [165349.330274] [] __alloc_pages_slowpath+0x6b6/0x724 [165349.330276] [] __alloc_pages_nodemask+0x405/0x420 [165349.330277] [] alloc_pages_current+0x98/0x110 [165349.330279] [] __get_free_pages+0xe/0x40 [165349.330280] [] swiotlb_alloc_coherent+0x5e/0x150 [165349.330282] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165349.330288] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165349.330293] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165349.330298] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165349.330299] [] ? dma_pool_free+0xa7/0xd0 [165349.330303] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165349.330307] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165349.330311] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165349.330317] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165349.330319] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165349.330322] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165349.330326] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165349.330328] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165349.330329] [] process_one_work+0x17a/0x440 [165349.330330] [] worker_thread+0x126/0x3c0 [165349.330332] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165349.330333] [] kthread+0xcf/0xe0 [165349.330334] [] ? insert_kthread_work+0x40/0x40 [165349.330335] [] ret_from_fork+0x58/0x90 [165349.330336] [] ? insert_kthread_work+0x40/0x40 [165349.330336] Mem-Info: [165349.330340] active_anon:6 inactive_anon:0 isolated_anon:0 active_file:59 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:35 unstable:0 slab_reclaimable:16083 slab_unreclaimable:564340 mapped:1550 shmem:0 pagetables:1560 bounce:0 free:39864 free_pcp:0 free_cma:0 [165349.330343] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165349.330344] lowmem_reserve[]: 0 2580 15620 15620 [165349.330348] Node 0 DMA32 free:59612kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165349.330349] lowmem_reserve[]: 0 0 13040 13040 [165349.330353] Node 0 Normal free:36840kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:156kB mapped:1508kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101760kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165349.330353] lowmem_reserve[]: 0 0 0 0 [165349.330357] Node 1 Normal free:47424kB min:45728kB low:57160kB high:68592kB active_anon:24kB inactive_anon:140kB active_file:356kB inactive_file:128kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4352kB shmem:0kB slab_reclaimable:33400kB slab_unreclaimable:958740kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165349.330358] lowmem_reserve[]: 0 0 0 0 [165349.330364] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165349.330368] Node 0 DMA32: 90*4kB (UEM) 2346*8kB (UEM) 1293*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59624kB [165349.330372] Node 0 Normal: 184*4kB (UEM) 176*8kB (UEM) 212*16kB (UEM) 286*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37152kB [165349.330377] Node 1 Normal: 521*4kB (UEM) 379*8kB (UEM) 217*16kB (UEM) 447*32kB (UEM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47532kB [165349.330377] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165349.330378] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165349.330379] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165349.330379] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165349.330379] 1848 total pagecache pages [165349.330380] 70 pages in swap cache [165349.330380] Swap cache stats: add 304510, delete 304440, find 90689/146067 [165349.330381] Free swap = 16247876kB [165349.330381] Total swap = 16319484kB [165349.330382] 8369063 pages RAM [165349.330382] 0 pages HighMem/MovableOnly [165349.330382] 241261 pages reserved [165350.664704] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165350.664706] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165350.664706] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165350.664712] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165350.664714] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165350.664715] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165350.664716] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165350.664716] Call Trace: [165350.664720] [] dump_stack+0x19/0x1b [165350.664723] [] warn_alloc_failed+0x110/0x180 [165350.664724] [] __alloc_pages_slowpath+0x6b6/0x724 [165350.664725] [] __alloc_pages_nodemask+0x405/0x420 [165350.664728] [] dma_generic_alloc_coherent+0x8f/0x140 [165350.664729] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165350.664742] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165350.664747] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165350.664752] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165350.664757] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165350.664761] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165350.664765] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165350.664772] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165350.664775] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165350.664780] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165350.664784] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165350.664786] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165350.664787] [] process_one_work+0x17a/0x440 [165350.664789] [] worker_thread+0x126/0x3c0 [165350.664790] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165350.664791] [] kthread+0xcf/0xe0 [165350.664793] [] ? insert_kthread_work+0x40/0x40 [165350.664794] [] ret_from_fork+0x58/0x90 [165350.664795] [] ? insert_kthread_work+0x40/0x40 [165350.664795] Mem-Info: [165350.664800] active_anon:62 inactive_anon:45 isolated_anon:0 active_file:71 inactive_file:358 isolated_file:0 unevictable:6774 dirty:0 writeback:80 unstable:0 slab_reclaimable:16083 slab_unreclaimable:564352 mapped:1600 shmem:4 pagetables:1560 bounce:0 free:39075 free_pcp:4 free_cma:0 [165350.664804] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165350.664805] lowmem_reserve[]: 0 2580 15620 15620 [165350.664809] Node 0 DMA32 free:59496kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165350.664810] lowmem_reserve[]: 0 0 13040 13040 [165350.664813] Node 0 Normal free:36840kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:156kB mapped:1508kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101792kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165350.664814] lowmem_reserve[]: 0 0 0 0 [165350.664818] Node 1 Normal free:44384kB min:45728kB low:57160kB high:68592kB active_anon:240kB inactive_anon:448kB active_file:400kB inactive_file:1620kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:124kB mapped:4552kB shmem:16kB slab_reclaimable:33400kB slab_unreclaimable:958756kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165350.664819] lowmem_reserve[]: 0 0 0 0 [165350.664824] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165350.664829] Node 0 DMA32: 70*4kB (UEM) 2346*8kB (UEM) 1293*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59544kB [165350.664833] Node 0 Normal: 172*4kB (UEM) 142*8kB (UEM) 205*16kB (UEM) 286*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36720kB [165350.664837] Node 1 Normal: 445*4kB (UEM) 218*8kB (UEM) 219*16kB (UEM) 445*32kB (UEM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45908kB [165350.664838] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165350.664839] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165350.664839] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165350.664840] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165350.664840] 2260 total pagecache pages [165350.664841] 155 pages in swap cache [165350.664842] Swap cache stats: add 305263, delete 305108, find 90915/146455 [165350.664842] Free swap = 16247956kB [165350.664842] Total swap = 16319484kB [165350.664843] 8369063 pages RAM [165350.664843] 0 pages HighMem/MovableOnly [165350.664844] 241261 pages reserved [165350.664890] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165350.664892] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165350.664892] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165350.664895] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165350.664896] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165350.664897] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165350.664898] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165350.664898] Call Trace: [165350.664900] [] dump_stack+0x19/0x1b [165350.664902] [] warn_alloc_failed+0x110/0x180 [165350.664903] [] ? drain_pages+0xb0/0xb0 [165350.664904] [] __alloc_pages_slowpath+0x6b6/0x724 [165350.664906] [] __alloc_pages_nodemask+0x405/0x420 [165350.664907] [] alloc_pages_current+0x98/0x110 [165350.664909] [] __get_free_pages+0xe/0x40 [165350.664911] [] swiotlb_alloc_coherent+0x5e/0x150 [165350.664912] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165350.664918] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165350.664923] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165350.664928] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165350.664932] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165350.664935] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165350.664940] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165350.664945] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165350.664947] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165350.664951] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165350.664954] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165350.664956] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165350.664958] [] process_one_work+0x17a/0x440 [165350.664959] [] worker_thread+0x126/0x3c0 [165350.664960] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165350.664961] [] kthread+0xcf/0xe0 [165350.664963] [] ? insert_kthread_work+0x40/0x40 [165350.664964] [] ret_from_fork+0x58/0x90 [165350.664965] [] ? insert_kthread_work+0x40/0x40 [165350.664965] Mem-Info: [165350.664969] active_anon:62 inactive_anon:45 isolated_anon:0 active_file:71 inactive_file:358 isolated_file:0 unevictable:6774 dirty:0 writeback:80 unstable:0 slab_reclaimable:16083 slab_unreclaimable:564352 mapped:1600 shmem:4 pagetables:1560 bounce:0 free:39165 free_pcp:0 free_cma:0 [165350.664972] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165350.664973] lowmem_reserve[]: 0 2580 15620 15620 [165350.664977] Node 0 DMA32 free:59496kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165350.664978] lowmem_reserve[]: 0 0 13040 13040 [165350.664982] Node 0 Normal free:36840kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:156kB mapped:1508kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101792kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165350.664983] lowmem_reserve[]: 0 0 0 0 [165350.664986] Node 1 Normal free:44744kB min:45728kB low:57160kB high:68592kB active_anon:240kB inactive_anon:448kB active_file:400kB inactive_file:1620kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:124kB mapped:4552kB shmem:16kB slab_reclaimable:33400kB slab_unreclaimable:958756kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165350.664987] lowmem_reserve[]: 0 0 0 0 [165350.664993] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165350.664997] Node 0 DMA32: 70*4kB (UEM) 2346*8kB (UEM) 1293*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59544kB [165350.665001] Node 0 Normal: 172*4kB (UEM) 142*8kB (UEM) 205*16kB (UEM) 286*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36720kB [165350.665006] Node 1 Normal: 445*4kB (UEM) 220*8kB (UEM) 219*16kB (UEM) 445*32kB (UEM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45924kB [165350.665006] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165350.665007] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165350.665007] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165350.665008] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165350.665008] 2260 total pagecache pages [165350.665009] 155 pages in swap cache [165350.665009] Swap cache stats: add 305263, delete 305108, find 90915/146455 [165350.665009] Free swap = 16247956kB [165350.665010] Total swap = 16319484kB [165350.665010] 8369063 pages RAM [165350.665011] 0 pages HighMem/MovableOnly [165350.665011] 241261 pages reserved [165353.270801] warn_alloc_failed: 2 callbacks suppressed [165353.270802] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165353.270804] CPU: 1 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165353.270804] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165353.270811] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165353.270813] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165353.270814] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165353.270815] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165353.270815] Call Trace: [165353.270820] [] dump_stack+0x19/0x1b [165353.270823] [] warn_alloc_failed+0x110/0x180 [165353.270824] [] __alloc_pages_slowpath+0x6b6/0x724 [165353.270826] [] __alloc_pages_nodemask+0x405/0x420 [165353.270829] [] dma_generic_alloc_coherent+0x8f/0x140 [165353.270831] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165353.270846] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165353.270851] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165353.270857] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165353.270860] [] ? dma_pool_free+0xa7/0xd0 [165353.270864] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165353.270868] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165353.270873] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165353.270880] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165353.270882] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165353.270888] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165353.270893] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165353.270895] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165353.270897] [] process_one_work+0x17a/0x440 [165353.270898] [] worker_thread+0x126/0x3c0 [165353.270900] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165353.270901] [] kthread+0xcf/0xe0 [165353.270902] [] ? insert_kthread_work+0x40/0x40 [165353.270904] [] ret_from_fork+0x58/0x90 [165353.270905] [] ? insert_kthread_work+0x40/0x40 [165353.270906] Mem-Info: [165353.270910] active_anon:74 inactive_anon:8 isolated_anon:0 active_file:98 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:119 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564360 mapped:1621 shmem:2 pagetables:1560 bounce:0 free:39252 free_pcp:0 free_cma:0 [165353.270914] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165353.270916] lowmem_reserve[]: 0 2580 15620 15620 [165353.270919] Node 0 DMA32 free:59484kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:184kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:14980 all_unreclaimable? no [165353.270920] lowmem_reserve[]: 0 0 13040 13040 [165353.270924] Node 0 Normal free:36144kB min:37012kB low:46264kB high:55516kB active_anon:44kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:200kB mapped:1616kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101808kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6386 all_unreclaimable? yes [165353.270925] lowmem_reserve[]: 0 0 0 0 [165353.270928] Node 1 Normal free:45800kB min:45728kB low:57160kB high:68592kB active_anon:248kB inactive_anon:428kB active_file:308kB inactive_file:176kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:236kB mapped:4524kB shmem:8kB slab_reclaimable:33432kB slab_unreclaimable:958772kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:168 all_unreclaimable? no [165353.270929] lowmem_reserve[]: 0 0 0 0 [165353.270934] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165353.270939] Node 0 DMA32: 81*4kB (UEM) 2346*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59572kB [165353.270943] Node 0 Normal: 264*4kB (UEM) 178*8kB (UEM) 204*16kB (UEM) 285*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37328kB [165353.270948] Node 1 Normal: 525*4kB (EM) 364*8kB (UEM) 217*16kB (UEM) 444*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47332kB [165353.270949] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165353.270949] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165353.270950] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165353.270950] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165353.270951] 2040 total pagecache pages [165353.270951] 80 pages in swap cache [165353.270952] Swap cache stats: add 305814, delete 305734, find 91076/146746 [165353.270953] Free swap = 16247872kB [165353.270953] Total swap = 16319484kB [165353.270954] 8369063 pages RAM [165353.270954] 0 pages HighMem/MovableOnly [165353.270954] 241261 pages reserved [165353.271076] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165353.271078] CPU: 1 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165353.271078] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165353.271087] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165353.271088] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165353.271089] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165353.271091] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165353.271091] Call Trace: [165353.271094] [] dump_stack+0x19/0x1b [165353.271096] [] warn_alloc_failed+0x110/0x180 [165353.271097] [] ? drain_pages+0xb0/0xb0 [165353.271098] [] __alloc_pages_slowpath+0x6b6/0x724 [165353.271099] [] __alloc_pages_nodemask+0x405/0x420 [165353.271101] [] alloc_pages_current+0x98/0x110 [165353.271103] [] __get_free_pages+0xe/0x40 [165353.271105] [] swiotlb_alloc_coherent+0x5e/0x150 [165353.271106] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165353.271113] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165353.271118] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165353.271123] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165353.271124] [] ? dma_pool_free+0xa7/0xd0 [165353.271129] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165353.271132] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165353.271137] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165353.271142] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165353.271144] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165353.271147] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165353.271151] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165353.271153] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165353.271154] [] process_one_work+0x17a/0x440 [165353.271156] [] worker_thread+0x126/0x3c0 [165353.271157] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165353.271158] [] kthread+0xcf/0xe0 [165353.271159] [] ? insert_kthread_work+0x40/0x40 [165353.271160] [] ret_from_fork+0x58/0x90 [165353.271161] [] ? insert_kthread_work+0x40/0x40 [165353.271162] Mem-Info: [165353.271166] active_anon:74 inactive_anon:8 isolated_anon:0 active_file:98 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:119 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564360 mapped:1621 shmem:2 pagetables:1560 bounce:0 free:39252 free_pcp:0 free_cma:0 [165353.271169] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165353.271170] lowmem_reserve[]: 0 2580 15620 15620 [165353.271174] Node 0 DMA32 free:59484kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:184kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:40kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:15006 all_unreclaimable? no [165353.271175] lowmem_reserve[]: 0 0 13040 13040 [165353.271179] Node 0 Normal free:36144kB min:37012kB low:46264kB high:55516kB active_anon:44kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:200kB mapped:1616kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101808kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6386 all_unreclaimable? yes [165353.271180] lowmem_reserve[]: 0 0 0 0 [165353.271184] Node 1 Normal free:45800kB min:45728kB low:57160kB high:68592kB active_anon:248kB inactive_anon:428kB active_file:308kB inactive_file:176kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:236kB mapped:4524kB shmem:8kB slab_reclaimable:33432kB slab_unreclaimable:958772kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:501 all_unreclaimable? no [165353.271185] lowmem_reserve[]: 0 0 0 0 [165353.271190] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165353.271194] Node 0 DMA32: 81*4kB (UEM) 2346*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59572kB [165353.271199] Node 0 Normal: 264*4kB (UEM) 178*8kB (UEM) 204*16kB (UEM) 285*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37328kB [165353.271203] Node 1 Normal: 525*4kB (EM) 364*8kB (UEM) 217*16kB (UEM) 444*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47332kB [165353.271204] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165353.271204] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165353.271205] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165353.271205] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165353.271206] 2040 total pagecache pages [165353.271206] 80 pages in swap cache [165353.271207] Swap cache stats: add 305814, delete 305734, find 91076/146746 [165353.271207] Free swap = 16247872kB [165353.271207] Total swap = 16319484kB [165353.271208] 8369063 pages RAM [165353.271208] 0 pages HighMem/MovableOnly [165353.271208] 241261 pages reserved [165353.339137] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165353.339138] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165353.339139] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165353.339143] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165353.339145] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165353.339146] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165353.339147] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165353.339147] Call Trace: [165353.339149] [] dump_stack+0x19/0x1b [165353.339151] [] warn_alloc_failed+0x110/0x180 [165353.339153] [] __alloc_pages_slowpath+0x6b6/0x724 [165353.339154] [] __alloc_pages_nodemask+0x405/0x420 [165353.339156] [] dma_generic_alloc_coherent+0x8f/0x140 [165353.339158] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165353.339165] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165353.339171] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165353.339176] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165353.339178] [] ? dma_pool_free+0xa7/0xd0 [165353.339182] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165353.339186] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165353.339191] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165353.339196] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165353.339198] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165353.339202] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165353.339206] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165353.339208] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165353.339209] [] process_one_work+0x17a/0x440 [165353.339211] [] worker_thread+0x126/0x3c0 [165353.339212] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165353.339213] [] kthread+0xcf/0xe0 [165353.339214] [] ? insert_kthread_work+0x40/0x40 [165353.339215] [] ret_from_fork+0x58/0x90 [165353.339216] [] ? insert_kthread_work+0x40/0x40 [165353.339217] Mem-Info: [165353.339221] active_anon:11 inactive_anon:60 isolated_anon:0 active_file:0 inactive_file:15 isolated_file:0 unevictable:6774 dirty:0 writeback:166 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564364 mapped:1558 shmem:0 pagetables:1560 bounce:0 free:39764 free_pcp:0 free_cma:0 [165353.339224] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165353.339226] lowmem_reserve[]: 0 2580 15620 15620 [165353.339229] Node 0 DMA32 free:59552kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:36kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:44kB mapped:336kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165353.339230] lowmem_reserve[]: 0 0 13040 13040 [165353.339234] Node 0 Normal free:36832kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:212kB mapped:1512kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101808kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6879 all_unreclaimable? yes [165353.339235] lowmem_reserve[]: 0 0 0 0 [165353.339239] Node 1 Normal free:47092kB min:45728kB low:57160kB high:68592kB active_anon:36kB inactive_anon:492kB active_file:256kB inactive_file:308kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:408kB mapped:4384kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958788kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165353.339240] lowmem_reserve[]: 0 0 0 0 [165353.339245] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165353.339250] Node 0 DMA32: 89*4kB (UE) 2344*8kB (UE) 1292*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59556kB [165353.339255] Node 0 Normal: 264*4kB (UEM) 178*8kB (UEM) 204*16kB (UEM) 284*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37296kB [165353.339259] Node 1 Normal: 514*4kB (EM) 366*8kB (UEM) 219*16kB (UEM) 441*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47240kB [165353.339260] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165353.339260] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165353.339261] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165353.339261] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165353.339262] 1895 total pagecache pages [165353.339262] 104 pages in swap cache [165353.339263] Swap cache stats: add 305838, delete 305734, find 91076/146746 [165353.339263] Free swap = 16247872kB [165353.339264] Total swap = 16319484kB [165353.339264] 8369063 pages RAM [165353.339265] 0 pages HighMem/MovableOnly [165353.339265] 241261 pages reserved [165353.339398] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165353.339399] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165353.339400] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165353.339402] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165353.339404] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165353.339404] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165353.339405] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165353.339406] Call Trace: [165353.339407] [] dump_stack+0x19/0x1b [165353.339409] [] warn_alloc_failed+0x110/0x180 [165353.339410] [] ? drain_pages+0xb0/0xb0 [165353.339411] [] __alloc_pages_slowpath+0x6b6/0x724 [165353.339413] [] __alloc_pages_nodemask+0x405/0x420 [165353.339414] [] alloc_pages_current+0x98/0x110 [165353.339416] [] __get_free_pages+0xe/0x40 [165353.339417] [] swiotlb_alloc_coherent+0x5e/0x150 [165353.339419] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165353.339425] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165353.339430] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165353.339435] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165353.339436] [] ? dma_pool_free+0xa7/0xd0 [165353.339441] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165353.339444] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165353.339449] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165353.339454] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165353.339456] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165353.339459] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165353.339463] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165353.339465] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165353.339467] [] process_one_work+0x17a/0x440 [165353.339468] [] worker_thread+0x126/0x3c0 [165353.339469] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165353.339470] [] kthread+0xcf/0xe0 [165353.339471] [] ? insert_kthread_work+0x40/0x40 [165353.339473] [] ret_from_fork+0x58/0x90 [165353.339474] [] ? insert_kthread_work+0x40/0x40 [165353.339474] Mem-Info: [165353.339478] active_anon:11 inactive_anon:60 isolated_anon:0 active_file:0 inactive_file:15 isolated_file:0 unevictable:6774 dirty:0 writeback:166 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564364 mapped:1558 shmem:0 pagetables:1560 bounce:0 free:39764 free_pcp:0 free_cma:0 [165353.339481] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165353.339482] lowmem_reserve[]: 0 2580 15620 15620 [165353.339486] Node 0 DMA32 free:59552kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:36kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:44kB mapped:336kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165353.339487] lowmem_reserve[]: 0 0 13040 13040 [165353.339491] Node 0 Normal free:36832kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:212kB mapped:1512kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101808kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6879 all_unreclaimable? yes [165353.339492] lowmem_reserve[]: 0 0 0 0 [165353.339496] Node 1 Normal free:47092kB min:45728kB low:57160kB high:68592kB active_anon:36kB inactive_anon:492kB active_file:256kB inactive_file:308kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:408kB mapped:4384kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958788kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:15 all_unreclaimable? no [165353.339497] lowmem_reserve[]: 0 0 0 0 [165353.339502] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165353.339506] Node 0 DMA32: 89*4kB (UE) 2344*8kB (UE) 1292*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59556kB [165353.339510] Node 0 Normal: 264*4kB (UEM) 178*8kB (UEM) 204*16kB (UEM) 284*32kB (UEM) 313*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37296kB [165353.339515] Node 1 Normal: 514*4kB (EM) 366*8kB (UEM) 219*16kB (UEM) 441*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47240kB [165353.339515] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165353.339516] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165353.339516] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165353.339517] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165353.339517] 1895 total pagecache pages [165353.339518] 104 pages in swap cache [165353.339518] Swap cache stats: add 305838, delete 305734, find 91076/146746 [165353.339518] Free swap = 16247872kB [165353.339519] Total swap = 16319484kB [165353.339519] 8369063 pages RAM [165353.339520] 0 pages HighMem/MovableOnly [165353.339520] 241261 pages reserved [165355.327540] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165355.327542] CPU: 19 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165355.327542] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165355.327547] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165355.327548] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165355.327549] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165355.327550] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165355.327551] Call Trace: [165355.327554] [] dump_stack+0x19/0x1b [165355.327557] [] warn_alloc_failed+0x110/0x180 [165355.327558] [] __alloc_pages_slowpath+0x6b6/0x724 [165355.327559] [] __alloc_pages_nodemask+0x405/0x420 [165355.327561] [] dma_generic_alloc_coherent+0x8f/0x140 [165355.327563] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165355.327573] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165355.327578] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165355.327584] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165355.327586] [] ? dma_pool_free+0xa7/0xd0 [165355.327590] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165355.327594] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165355.327599] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165355.327605] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165355.327607] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165355.327611] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165355.327615] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165355.327617] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165355.327619] [] process_one_work+0x17a/0x440 [165355.327620] [] worker_thread+0x126/0x3c0 [165355.327622] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165355.327623] [] kthread+0xcf/0xe0 [165355.327624] [] ? insert_kthread_work+0x40/0x40 [165355.327625] [] ret_from_fork+0x58/0x90 [165355.327626] [] ? insert_kthread_work+0x40/0x40 [165355.327627] Mem-Info: [165355.327631] active_anon:4 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:116 isolated_file:0 unevictable:6774 dirty:0 writeback:101 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564376 mapped:1612 shmem:0 pagetables:1560 bounce:0 free:39456 free_pcp:0 free_cma:0 [165355.327635] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165355.327636] lowmem_reserve[]: 0 2580 15620 15620 [165355.327640] Node 0 DMA32 free:59488kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:60kB mapped:328kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165355.327641] lowmem_reserve[]: 0 0 13040 13040 [165355.327644] Node 0 Normal free:36944kB min:37012kB low:46264kB high:55516kB active_anon:16kB inactive_anon:0kB active_file:0kB inactive_file:80kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:236kB mapped:1544kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101840kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1254 all_unreclaimable? yes [165355.327645] lowmem_reserve[]: 0 0 0 0 [165355.327649] Node 1 Normal free:45812kB min:45728kB low:57160kB high:68592kB active_anon:24kB inactive_anon:348kB active_file:64kB inactive_file:404kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:108kB mapped:4576kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7 all_unreclaimable? no [165355.327649] lowmem_reserve[]: 0 0 0 0 [165355.327655] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165355.327659] Node 0 DMA32: 85*4kB (UE) 2347*8kB (UEM) 1292*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59500kB [165355.327664] Node 0 Normal: 254*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 283*32kB (UEM) 314*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37352kB [165355.327668] Node 1 Normal: 427*4kB (UEM) 363*8kB (UEM) 217*16kB (UEM) 443*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46900kB [165355.327669] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165355.327669] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165355.327670] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165355.327671] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165355.327671] 1909 total pagecache pages [165355.327672] 106 pages in swap cache [165355.327672] Swap cache stats: add 306421, delete 306315, find 91263/147067 [165355.327673] Free swap = 16247872kB [165355.327673] Total swap = 16319484kB [165355.327674] 8369063 pages RAM [165355.327674] 0 pages HighMem/MovableOnly [165355.327674] 241261 pages reserved [165355.327848] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165355.327849] CPU: 19 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165355.327850] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165355.327852] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165355.327854] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165355.327855] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165355.327856] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165355.327856] Call Trace: [165355.327858] [] dump_stack+0x19/0x1b [165355.327859] [] warn_alloc_failed+0x110/0x180 [165355.327860] [] ? drain_pages+0xb0/0xb0 [165355.327862] [] __alloc_pages_slowpath+0x6b6/0x724 [165355.327863] [] __alloc_pages_nodemask+0x405/0x420 [165355.327864] [] alloc_pages_current+0x98/0x110 [165355.327866] [] __get_free_pages+0xe/0x40 [165355.327868] [] swiotlb_alloc_coherent+0x5e/0x150 [165355.327869] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165355.327876] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165355.327881] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165355.327885] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165355.327887] [] ? dma_pool_free+0xa7/0xd0 [165355.327891] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165355.327895] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165355.327899] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165355.327904] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165355.327906] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165355.327910] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165355.327913] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165355.327915] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165355.327917] [] process_one_work+0x17a/0x440 [165355.327918] [] worker_thread+0x126/0x3c0 [165355.327919] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165355.327920] [] kthread+0xcf/0xe0 [165355.327922] [] ? insert_kthread_work+0x40/0x40 [165355.327923] [] ret_from_fork+0x58/0x90 [165355.327924] [] ? insert_kthread_work+0x40/0x40 [165355.327924] Mem-Info: [165355.327928] active_anon:4 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:116 isolated_file:0 unevictable:6774 dirty:0 writeback:101 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564376 mapped:1612 shmem:0 pagetables:1560 bounce:0 free:39456 free_pcp:0 free_cma:0 [165355.327931] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165355.327932] lowmem_reserve[]: 0 2580 15620 15620 [165355.327936] Node 0 DMA32 free:59488kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:60kB mapped:328kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no [165355.327937] lowmem_reserve[]: 0 0 13040 13040 [165355.327941] Node 0 Normal free:36944kB min:37012kB low:46264kB high:55516kB active_anon:16kB inactive_anon:0kB active_file:0kB inactive_file:80kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:236kB mapped:1544kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101840kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1254 all_unreclaimable? yes [165355.327941] lowmem_reserve[]: 0 0 0 0 [165355.327945] Node 1 Normal free:45812kB min:45728kB low:57160kB high:68592kB active_anon:24kB inactive_anon:348kB active_file:64kB inactive_file:404kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:108kB mapped:4576kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:53 all_unreclaimable? no [165355.327946] lowmem_reserve[]: 0 0 0 0 [165355.327951] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165355.327955] Node 0 DMA32: 85*4kB (UE) 2347*8kB (UEM) 1292*16kB (UEM) 460*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59500kB [165355.327960] Node 0 Normal: 254*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 283*32kB (UEM) 314*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37352kB [165355.327964] Node 1 Normal: 427*4kB (UEM) 363*8kB (UEM) 217*16kB (UEM) 443*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46900kB [165355.327965] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165355.327965] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165355.327966] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165355.327967] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165355.327967] 1909 total pagecache pages [165355.327967] 106 pages in swap cache [165355.327968] Swap cache stats: add 306421, delete 306315, find 91263/147067 [165355.327968] Free swap = 16247872kB [165355.327969] Total swap = 16319484kB [165355.327970] 8369063 pages RAM [165355.327970] 0 pages HighMem/MovableOnly [165355.327970] 241261 pages reserved [165355.343324] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165355.343326] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165355.343326] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165355.343330] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165355.343332] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165355.343333] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165355.343334] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165355.343334] Call Trace: [165355.343337] [] dump_stack+0x19/0x1b [165355.343339] [] warn_alloc_failed+0x110/0x180 [165355.343340] [] __alloc_pages_slowpath+0x6b6/0x724 [165355.343341] [] __alloc_pages_nodemask+0x405/0x420 [165355.343343] [] dma_generic_alloc_coherent+0x8f/0x140 [165355.343345] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165355.343353] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165355.343358] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165355.343363] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165355.343365] [] ? dma_pool_free+0xa7/0xd0 [165355.343369] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165355.343373] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165355.343378] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165355.343384] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165355.343386] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165355.343390] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165355.343394] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165355.343396] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165355.343398] [] process_one_work+0x17a/0x440 [165355.343399] [] worker_thread+0x126/0x3c0 [165355.343401] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165355.343402] [] kthread+0xcf/0xe0 [165355.343403] [] ? insert_kthread_work+0x40/0x40 [165355.343404] [] ret_from_fork+0x58/0x90 [165355.343405] [] ? insert_kthread_work+0x40/0x40 [165355.343406] Mem-Info: [165355.343410] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:22 inactive_file:47 isolated_file:0 unevictable:6774 dirty:0 writeback:97 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564376 mapped:1551 shmem:0 pagetables:1560 bounce:0 free:39643 free_pcp:27 free_cma:0 [165355.343414] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165355.343415] lowmem_reserve[]: 0 2580 15620 15620 [165355.343419] Node 0 DMA32 free:59540kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:56kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:60kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165355.343420] lowmem_reserve[]: 0 0 13040 13040 [165355.343424] Node 0 Normal free:36752kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:228kB mapped:1544kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101840kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165355.343424] lowmem_reserve[]: 0 0 0 0 [165355.343428] Node 1 Normal free:46700kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:308kB active_file:196kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:100kB mapped:4320kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:108kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165355.343429] lowmem_reserve[]: 0 0 0 0 [165355.343434] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165355.343439] Node 0 DMA32: 96*4kB (UEM) 2348*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59584kB [165355.343443] Node 0 Normal: 254*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 283*32kB (UEM) 314*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37352kB [165355.343447] Node 1 Normal: 405*4kB (UEM) 364*8kB (UEM) 219*16kB (UEM) 441*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46788kB [165355.343448] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165355.343449] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165355.343449] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165355.343450] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165355.343450] 1828 total pagecache pages [165355.343451] 92 pages in swap cache [165355.343451] Swap cache stats: add 306432, delete 306340, find 91266/147073 [165355.343452] Free swap = 16247876kB [165355.343452] Total swap = 16319484kB [165355.343453] 8369063 pages RAM [165355.343453] 0 pages HighMem/MovableOnly [165355.343453] 241261 pages reserved [165355.343669] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165355.343670] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165355.343670] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165355.343673] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165355.343674] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165355.343675] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165355.343676] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165355.343676] Call Trace: [165355.343678] [] dump_stack+0x19/0x1b [165355.343680] [] warn_alloc_failed+0x110/0x180 [165355.343681] [] ? drain_pages+0xb0/0xb0 [165355.343682] [] __alloc_pages_slowpath+0x6b6/0x724 [165355.343684] [] __alloc_pages_nodemask+0x405/0x420 [165355.343685] [] alloc_pages_current+0x98/0x110 [165355.343687] [] __get_free_pages+0xe/0x40 [165355.343688] [] swiotlb_alloc_coherent+0x5e/0x150 [165355.343690] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165355.343696] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165355.343701] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165355.343706] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165355.343707] [] ? dma_pool_free+0xa7/0xd0 [165355.343712] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165355.343715] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165355.343720] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165355.343725] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165355.343727] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165355.343731] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165355.343734] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165355.343736] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165355.343738] [] process_one_work+0x17a/0x440 [165355.343739] [] worker_thread+0x126/0x3c0 [165355.343740] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165355.343741] [] kthread+0xcf/0xe0 [165355.343743] [] ? insert_kthread_work+0x40/0x40 [165355.343744] [] ret_from_fork+0x58/0x90 [165355.343745] [] ? insert_kthread_work+0x40/0x40 [165355.343745] Mem-Info: [165355.343749] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:22 inactive_file:47 isolated_file:0 unevictable:6774 dirty:0 writeback:97 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564376 mapped:1551 shmem:0 pagetables:1560 bounce:0 free:39643 free_pcp:0 free_cma:0 [165355.343752] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165355.343753] lowmem_reserve[]: 0 2580 15620 15620 [165355.343757] Node 0 DMA32 free:59540kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:56kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:60kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165355.343758] lowmem_reserve[]: 0 0 13040 13040 [165355.343761] Node 0 Normal free:36752kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:228kB mapped:1544kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101840kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165355.343762] lowmem_reserve[]: 0 0 0 0 [165355.343766] Node 1 Normal free:46700kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:308kB active_file:196kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:100kB mapped:4320kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1 all_unreclaimable? no [165355.343767] lowmem_reserve[]: 0 0 0 0 [165355.343772] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165355.343776] Node 0 DMA32: 96*4kB (UEM) 2348*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59584kB [165355.343781] Node 0 Normal: 254*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 283*32kB (UEM) 314*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37352kB [165355.343785] Node 1 Normal: 432*4kB (UEM) 364*8kB (UEM) 219*16kB (UEM) 441*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46896kB [165355.343785] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165355.343786] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165355.343787] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165355.343787] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165355.343787] 1828 total pagecache pages [165355.343788] 92 pages in swap cache [165355.343788] Swap cache stats: add 306432, delete 306340, find 91266/147073 [165355.343789] Free swap = 16247876kB [165355.343789] Total swap = 16319484kB [165355.343790] 8369063 pages RAM [165355.343790] 0 pages HighMem/MovableOnly [165355.343790] 241261 pages reserved [165356.294171] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165356.294173] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165356.294174] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165356.294184] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165356.294186] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165356.294188] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165356.294189] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165356.294190] Call Trace: [165356.294196] [] dump_stack+0x19/0x1b [165356.294200] [] warn_alloc_failed+0x110/0x180 [165356.294203] [] __alloc_pages_slowpath+0x6b6/0x724 [165356.294205] [] __alloc_pages_nodemask+0x405/0x420 [165356.294209] [] dma_generic_alloc_coherent+0x8f/0x140 [165356.294212] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165356.294233] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165356.294239] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165356.294248] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165356.294250] [] ? dma_pool_free+0xa7/0xd0 [165356.294256] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165356.294261] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165356.294267] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165356.294275] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165356.294278] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165356.294287] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165356.294295] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165356.294298] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165356.294301] [] process_one_work+0x17a/0x440 [165356.294303] [] worker_thread+0x126/0x3c0 [165356.294305] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165356.294307] [] kthread+0xcf/0xe0 [165356.294309] [] ? insert_kthread_work+0x40/0x40 [165356.294312] [] ret_from_fork+0x58/0x90 [165356.294314] [] ? insert_kthread_work+0x40/0x40 [165356.294315] Mem-Info: [165356.294321] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:133 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:100 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564376 mapped:1561 shmem:0 pagetables:1560 bounce:0 free:39665 free_pcp:28 free_cma:0 [165356.294327] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165356.294329] lowmem_reserve[]: 0 2580 15620 15620 [165356.294335] Node 0 DMA32 free:59548kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:16kB active_file:52kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:64kB mapped:336kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:22 all_unreclaimable? no [165356.294336] lowmem_reserve[]: 0 0 13040 13040 [165356.294341] Node 0 Normal free:36620kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:248kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:228kB mapped:1528kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101840kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:483 all_unreclaimable? yes [165356.294343] lowmem_reserve[]: 0 0 0 0 [165356.294349] Node 1 Normal free:46912kB min:45728kB low:57160kB high:68592kB active_anon:20kB inactive_anon:184kB active_file:232kB inactive_file:384kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:108kB mapped:4380kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:201 all_unreclaimable? no [165356.294350] lowmem_reserve[]: 0 0 0 0 [165356.294358] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165356.294365] Node 0 DMA32: 87*4kB (UE) 2344*8kB (UEM) 1293*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165356.294372] Node 0 Normal: 256*4kB (UEM) 169*8kB (UEM) 208*16kB (UEM) 284*32kB (UEM) 314*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37320kB [165356.294379] Node 1 Normal: 478*4kB (UEM) 368*8kB (UEM) 220*16kB (UEM) 441*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47128kB [165356.294380] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165356.294381] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165356.294382] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165356.294383] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165356.294384] 1740 total pagecache pages [165356.294385] 82 pages in swap cache [165356.294386] Swap cache stats: add 306613, delete 306531, find 91319/147166 [165356.294386] Free swap = 16247872kB [165356.294386] Total swap = 16319484kB [165356.294388] 8369063 pages RAM [165356.294388] 0 pages HighMem/MovableOnly [165356.294388] 241261 pages reserved [165356.294591] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165356.294593] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165356.294593] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165356.294597] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165356.294599] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165356.294600] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165356.294602] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165356.294602] Call Trace: [165356.294604] [] dump_stack+0x19/0x1b [165356.294607] [] warn_alloc_failed+0x110/0x180 [165356.294608] [] ? drain_pages+0xb0/0xb0 [165356.294610] [] __alloc_pages_slowpath+0x6b6/0x724 [165356.294612] [] __alloc_pages_nodemask+0x405/0x420 [165356.294614] [] alloc_pages_current+0x98/0x110 [165356.294617] [] __get_free_pages+0xe/0x40 [165356.294620] [] swiotlb_alloc_coherent+0x5e/0x150 [165356.294622] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165356.294630] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165356.294636] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165356.294642] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165356.294644] [] ? dma_pool_free+0xa7/0xd0 [165356.294649] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165356.294654] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165356.294659] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165356.294666] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165356.294669] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165356.294673] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165356.294678] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165356.294681] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165356.294683] [] process_one_work+0x17a/0x440 [165356.294685] [] worker_thread+0x126/0x3c0 [165356.294687] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165356.294688] [] kthread+0xcf/0xe0 [165356.294690] [] ? insert_kthread_work+0x40/0x40 [165356.294692] [] ret_from_fork+0x58/0x90 [165356.294693] [] ? insert_kthread_work+0x40/0x40 [165356.294694] Mem-Info: [165356.294700] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:133 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:100 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564376 mapped:1561 shmem:0 pagetables:1560 bounce:0 free:39665 free_pcp:0 free_cma:0 [165356.294705] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165356.294706] lowmem_reserve[]: 0 2580 15620 15620 [165356.294712] Node 0 DMA32 free:59548kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:16kB active_file:52kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:64kB mapped:336kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:22 all_unreclaimable? no [165356.294713] lowmem_reserve[]: 0 0 13040 13040 [165356.294719] Node 0 Normal free:36620kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:248kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:228kB mapped:1528kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101840kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:483 all_unreclaimable? yes [165356.294720] lowmem_reserve[]: 0 0 0 0 [165356.294725] Node 1 Normal free:46912kB min:45728kB low:57160kB high:68592kB active_anon:20kB inactive_anon:184kB active_file:232kB inactive_file:384kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:108kB mapped:4380kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165356.294726] lowmem_reserve[]: 0 0 0 0 [165356.294734] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165356.294741] Node 0 DMA32: 87*4kB (UE) 2344*8kB (UEM) 1293*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165356.294748] Node 0 Normal: 256*4kB (UEM) 169*8kB (UEM) 208*16kB (UEM) 284*32kB (UEM) 314*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37320kB [165356.294755] Node 1 Normal: 503*4kB (UEM) 368*8kB (UEM) 220*16kB (UEM) 441*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47228kB [165356.294756] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165356.294757] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165356.294757] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165356.294758] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165356.294759] 1740 total pagecache pages [165356.294760] 85 pages in swap cache [165356.294761] Swap cache stats: add 306616, delete 306531, find 91319/147166 [165356.294761] Free swap = 16247872kB [165356.294761] Total swap = 16319484kB [165356.294762] 8369063 pages RAM [165356.294763] 0 pages HighMem/MovableOnly [165356.294763] 241261 pages reserved [165358.355894] warn_alloc_failed: 2 callbacks suppressed [165358.355895] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165358.355897] CPU: 3 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165358.355898] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165358.355911] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165358.355913] 00000000000080d0 00000000fa2ca73a ffff88073f657810 ffffffff816a3e2d [165358.355915] ffff88073f6578a0 ffffffff81188820 0000000000000000 00000000ffffffff [165358.355916] ffffffffffffff00 000080d000000000 ffff88073f657870 00000000fa2ca73a [165358.355916] Call Trace: [165358.355922] [] dump_stack+0x19/0x1b [165358.355926] [] warn_alloc_failed+0x110/0x180 [165358.355928] [] __alloc_pages_slowpath+0x6b6/0x724 [165358.355929] [] __alloc_pages_nodemask+0x405/0x420 [165358.355932] [] dma_generic_alloc_coherent+0x8f/0x140 [165358.355935] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165358.355956] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165358.355961] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165358.355967] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165358.355972] [] ? mlx4_ib_create_qp+0x350/0xdc0 [mlx4_ib] [165358.355976] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165358.355980] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165358.355984] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165358.355992] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165358.355994] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165358.356000] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165358.356005] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165358.356007] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165358.356009] [] process_one_work+0x17a/0x440 [165358.356011] [] worker_thread+0x126/0x3c0 [165358.356012] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165358.356013] [] kthread+0xcf/0xe0 [165358.356015] [] ? insert_kthread_work+0x40/0x40 [165358.356016] [] ret_from_fork+0x58/0x90 [165358.356017] [] ? insert_kthread_work+0x40/0x40 [165358.356018] Mem-Info: [165358.356022] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:137 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:131 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564376 mapped:1565 shmem:0 pagetables:1560 bounce:0 free:39694 free_pcp:0 free_cma:0 [165358.356027] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165358.356028] lowmem_reserve[]: 0 2580 15620 15620 [165358.356032] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:52kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:56kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165358.356033] lowmem_reserve[]: 0 0 13040 13040 [165358.356037] Node 0 Normal free:37172kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:208kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:260kB mapped:1520kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101840kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165358.356038] lowmem_reserve[]: 0 0 0 0 [165358.356041] Node 1 Normal free:46544kB min:45728kB low:57160kB high:68592kB active_anon:12kB inactive_anon:280kB active_file:288kB inactive_file:448kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:208kB mapped:4396kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165358.356042] lowmem_reserve[]: 0 0 0 0 [165358.356047] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165358.356052] Node 0 DMA32: 91*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59580kB [165358.356057] Node 0 Normal: 255*4kB (UEM) 170*8kB (UEM) 207*16kB (UEM) 282*32kB (UM) 314*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37244kB [165358.356061] Node 1 Normal: 457*4kB (UEM) 370*8kB (UEM) 219*16kB (UEM) 432*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46756kB [165358.356062] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165358.356063] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165358.356063] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165358.356064] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165358.356065] 1788 total pagecache pages [165358.356065] 134 pages in swap cache [165358.356066] Swap cache stats: add 307456, delete 307322, find 91548/147562 [165358.356066] Free swap = 16247876kB [165358.356067] Total swap = 16319484kB [165358.356068] 8369063 pages RAM [165358.356068] 0 pages HighMem/MovableOnly [165358.356068] 241261 pages reserved [165358.356141] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165358.356142] CPU: 3 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165358.356143] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165358.356145] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165358.356147] 00000000000080d0 00000000fa2ca73a ffff88073f6577c0 ffffffff816a3e2d [165358.356148] ffff88073f657850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165358.356149] ffffffffffffff00 000080d000000000 ffff88073f657820 00000000fa2ca73a [165358.356149] Call Trace: [165358.356151] [] dump_stack+0x19/0x1b [165358.356152] [] warn_alloc_failed+0x110/0x180 [165358.356154] [] ? drain_pages+0xb0/0xb0 [165358.356155] [] __alloc_pages_slowpath+0x6b6/0x724 [165358.356156] [] __alloc_pages_nodemask+0x405/0x420 [165358.356158] [] alloc_pages_current+0x98/0x110 [165358.356160] [] __get_free_pages+0xe/0x40 [165358.356163] [] swiotlb_alloc_coherent+0x5e/0x150 [165358.356164] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165358.356171] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165358.356176] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165358.356180] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165358.356184] [] ? mlx4_ib_create_qp+0x350/0xdc0 [mlx4_ib] [165358.356188] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165358.356192] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165358.356196] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165358.356202] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165358.356204] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165358.356207] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165358.356211] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165358.356213] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165358.356214] [] process_one_work+0x17a/0x440 [165358.356216] [] worker_thread+0x126/0x3c0 [165358.356217] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165358.356218] [] kthread+0xcf/0xe0 [165358.356219] [] ? insert_kthread_work+0x40/0x40 [165358.356220] [] ret_from_fork+0x58/0x90 [165358.356221] [] ? insert_kthread_work+0x40/0x40 [165358.356222] Mem-Info: [165358.356226] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:137 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:131 unstable:0 slab_reclaimable:16091 slab_unreclaimable:564376 mapped:1565 shmem:0 pagetables:1560 bounce:0 free:39694 free_pcp:0 free_cma:0 [165358.356229] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165358.356230] lowmem_reserve[]: 0 2580 15620 15620 [165358.356234] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:52kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:56kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165358.356235] lowmem_reserve[]: 0 0 13040 13040 [165358.356238] Node 0 Normal free:37172kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:208kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:260kB mapped:1520kB shmem:0kB slab_reclaimable:26648kB slab_unreclaimable:1101840kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165358.356239] lowmem_reserve[]: 0 0 0 0 [165358.356243] Node 1 Normal free:46544kB min:45728kB low:57160kB high:68592kB active_anon:12kB inactive_anon:280kB active_file:288kB inactive_file:448kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:208kB mapped:4396kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165358.356244] lowmem_reserve[]: 0 0 0 0 [165358.356249] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165358.356253] Node 0 DMA32: 91*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59580kB [165358.356257] Node 0 Normal: 255*4kB (UEM) 170*8kB (UEM) 207*16kB (UEM) 282*32kB (UM) 314*64kB (UM) 17*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37244kB [165358.356261] Node 1 Normal: 457*4kB (UEM) 370*8kB (UEM) 219*16kB (UEM) 432*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46756kB [165358.356262] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165358.356263] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165358.356263] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165358.356264] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165358.356264] 1788 total pagecache pages [165358.356264] 134 pages in swap cache [165358.356265] Swap cache stats: add 307456, delete 307322, find 91548/147562 [165358.356265] Free swap = 16247876kB [165358.356266] Total swap = 16319484kB [165358.356266] 8369063 pages RAM [165358.356266] 0 pages HighMem/MovableOnly [165358.356267] 241261 pages reserved [165360.306485] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165360.306486] CPU: 5 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165360.306487] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165360.306492] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165360.306493] 00000000000080d0 00000000fa2ca73a ffff88073f657810 ffffffff816a3e2d [165360.306494] ffff88073f6578a0 ffffffff81188820 0000000000000000 00000000ffffffff [165360.306495] ffffffffffffff00 000080d000000000 ffff88073f657870 00000000fa2ca73a [165360.306496] Call Trace: [165360.306499] [] dump_stack+0x19/0x1b [165360.306501] [] warn_alloc_failed+0x110/0x180 [165360.306502] [] __alloc_pages_slowpath+0x6b6/0x724 [165360.306504] [] __alloc_pages_nodemask+0x405/0x420 [165360.306506] [] dma_generic_alloc_coherent+0x8f/0x140 [165360.306508] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165360.306518] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165360.306523] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165360.306529] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165360.306531] [] ? dma_pool_free+0xa7/0xd0 [165360.306536] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165360.306540] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165360.306545] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165360.306551] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165360.306553] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165360.306557] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165360.306561] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165360.306563] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165360.306565] [] process_one_work+0x17a/0x440 [165360.306566] [] worker_thread+0x126/0x3c0 [165360.306568] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165360.306569] [] kthread+0xcf/0xe0 [165360.306570] [] ? insert_kthread_work+0x40/0x40 [165360.306571] [] ret_from_fork+0x58/0x90 [165360.306572] [] ? insert_kthread_work+0x40/0x40 [165360.306573] Mem-Info: [165360.306577] active_anon:53 inactive_anon:0 isolated_anon:0 active_file:47 inactive_file:125 isolated_file:0 unevictable:6774 dirty:0 writeback:115 unstable:0 slab_reclaimable:16099 slab_unreclaimable:564384 mapped:1614 shmem:2 pagetables:1560 bounce:0 free:39500 free_pcp:82 free_cma:0 [165360.306581] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165360.306582] lowmem_reserve[]: 0 2580 15620 15620 [165360.306586] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:104kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:52kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no [165360.306587] lowmem_reserve[]: 0 0 13040 13040 [165360.306590] Node 0 Normal free:36816kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:96kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:256kB mapped:1512kB shmem:4kB slab_reclaimable:26680kB slab_unreclaimable:1101872kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1186 all_unreclaimable? yes [165360.306591] lowmem_reserve[]: 0 0 0 0 [165360.306595] Node 1 Normal free:46124kB min:45728kB low:57160kB high:68592kB active_anon:236kB inactive_anon:256kB active_file:528kB inactive_file:480kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:152kB mapped:4604kB shmem:4kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:260kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:197 all_unreclaimable? no [165360.306595] lowmem_reserve[]: 0 0 0 0 [165360.306601] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165360.306605] Node 0 DMA32: 86*4kB (UEM) 2347*8kB (UEM) 1292*16kB (UE) 459*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59536kB [165360.306610] Node 0 Normal: 193*4kB (UEM) 171*8kB (UEM) 207*16kB (UEM) 283*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36972kB [165360.306614] Node 1 Normal: 464*4kB (UEM) 363*8kB (UEM) 218*16kB (UEM) 443*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47064kB [165360.306615] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165360.306616] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165360.306616] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165360.306617] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165360.306617] 1980 total pagecache pages [165360.306618] 121 pages in swap cache [165360.306618] Swap cache stats: add 308147, delete 308026, find 91752/147923 [165360.306619] Free swap = 16247872kB [165360.306619] Total swap = 16319484kB [165360.306620] 8369063 pages RAM [165360.306620] 0 pages HighMem/MovableOnly [165360.306620] 241261 pages reserved [165360.306741] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165360.306742] CPU: 5 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165360.306742] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165360.306745] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165360.306746] 00000000000080d0 00000000fa2ca73a ffff88073f6577c0 ffffffff816a3e2d [165360.306747] ffff88073f657850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165360.306748] ffffffffffffff00 000080d000000000 ffff88073f657820 00000000fa2ca73a [165360.306748] Call Trace: [165360.306750] [] dump_stack+0x19/0x1b [165360.306752] [] warn_alloc_failed+0x110/0x180 [165360.306753] [] ? drain_pages+0xb0/0xb0 [165360.306754] [] __alloc_pages_slowpath+0x6b6/0x724 [165360.306755] [] __alloc_pages_nodemask+0x405/0x420 [165360.306757] [] alloc_pages_current+0x98/0x110 [165360.306758] [] __get_free_pages+0xe/0x40 [165360.306760] [] swiotlb_alloc_coherent+0x5e/0x150 [165360.306762] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165360.306768] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165360.306773] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165360.306778] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165360.306779] [] ? dma_pool_free+0xa7/0xd0 [165360.306783] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165360.306787] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165360.306791] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165360.306796] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165360.306798] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165360.306802] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165360.306805] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165360.306807] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165360.306809] [] process_one_work+0x17a/0x440 [165360.306810] [] worker_thread+0x126/0x3c0 [165360.306811] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165360.306812] [] kthread+0xcf/0xe0 [165360.306814] [] ? insert_kthread_work+0x40/0x40 [165360.306815] [] ret_from_fork+0x58/0x90 [165360.306816] [] ? insert_kthread_work+0x40/0x40 [165360.306816] Mem-Info: [165360.306820] active_anon:53 inactive_anon:0 isolated_anon:0 active_file:53 inactive_file:133 isolated_file:0 unevictable:6774 dirty:0 writeback:115 unstable:0 slab_reclaimable:16099 slab_unreclaimable:564384 mapped:1614 shmem:2 pagetables:1560 bounce:0 free:39469 free_pcp:3 free_cma:0 [165360.306823] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165360.306824] lowmem_reserve[]: 0 2580 15620 15620 [165360.306828] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:104kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:52kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:19 all_unreclaimable? no [165360.306829] lowmem_reserve[]: 0 0 13040 13040 [165360.306833] Node 0 Normal free:36816kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:96kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:256kB mapped:1512kB shmem:4kB slab_reclaimable:26680kB slab_unreclaimable:1101872kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1186 all_unreclaimable? yes [165360.306834] lowmem_reserve[]: 0 0 0 0 [165360.306845] Node 1 Normal free:46000kB min:45728kB low:57160kB high:68592kB active_anon:236kB inactive_anon:264kB active_file:552kB inactive_file:512kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:152kB mapped:4604kB shmem:4kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:234 all_unreclaimable? no [165360.306846] lowmem_reserve[]: 0 0 0 0 [165360.306851] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165360.306855] Node 0 DMA32: 87*4kB (UEM) 2349*8kB (UEM) 1293*16kB (UEM) 459*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59572kB [165360.306859] Node 0 Normal: 193*4kB (UEM) 171*8kB (UEM) 207*16kB (UEM) 283*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36972kB [165360.306863] Node 1 Normal: 467*4kB (UEM) 363*8kB (UEM) 218*16kB (UEM) 443*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47076kB [165360.306864] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165360.306865] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165360.306865] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165360.306866] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165360.306866] 1980 total pagecache pages [165360.306867] 119 pages in swap cache [165360.306867] Swap cache stats: add 308149, delete 308030, find 91752/147923 [165360.306868] Free swap = 16247872kB [165360.306868] Total swap = 16319484kB [165360.306869] 8369063 pages RAM [165360.306869] 0 pages HighMem/MovableOnly [165360.306869] 241261 pages reserved [165360.367966] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165360.367969] CPU: 0 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165360.367970] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165360.367976] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165360.367978] 00000000000080d0 00000000fa2ca73a ffff88073f657810 ffffffff816a3e2d [165360.367980] ffff88073f6578a0 ffffffff81188820 0000000000000000 00000000ffffffff [165360.367981] ffffffffffffff00 000080d000000000 ffff88073f657870 00000000fa2ca73a [165360.367982] Call Trace: [165360.367985] [] dump_stack+0x19/0x1b [165360.367988] [] warn_alloc_failed+0x110/0x180 [165360.367990] [] __alloc_pages_slowpath+0x6b6/0x724 [165360.367993] [] __alloc_pages_nodemask+0x405/0x420 [165360.367995] [] dma_generic_alloc_coherent+0x8f/0x140 [165360.367998] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165360.368012] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165360.368018] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165360.368025] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165360.368028] [] ? dma_pool_free+0xa7/0xd0 [165360.368034] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165360.368039] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165360.368045] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165360.368053] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165360.368056] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165360.368063] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165360.368068] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165360.368072] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165360.368074] [] process_one_work+0x17a/0x440 [165360.368076] [] worker_thread+0x126/0x3c0 [165360.368078] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165360.368080] [] kthread+0xcf/0xe0 [165360.368082] [] ? insert_kthread_work+0x40/0x40 [165360.368084] [] ret_from_fork+0x58/0x90 [165360.368086] [] ? insert_kthread_work+0x40/0x40 [165360.368087] Mem-Info: [165360.368092] active_anon:8 inactive_anon:0 isolated_anon:0 active_file:20 inactive_file:46 isolated_file:0 unevictable:6774 dirty:0 writeback:122 unstable:0 slab_reclaimable:16107 slab_unreclaimable:564384 mapped:1543 shmem:0 pagetables:1560 bounce:0 free:39703 free_pcp:0 free_cma:0 [165360.368098] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165360.368099] lowmem_reserve[]: 0 2580 15620 15620 [165360.368105] Node 0 DMA32 free:59584kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165360.368107] lowmem_reserve[]: 0 0 13040 13040 [165360.368112] Node 0 Normal free:36800kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:292kB mapped:1508kB shmem:0kB slab_reclaimable:26712kB slab_unreclaimable:1101872kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165360.368114] lowmem_reserve[]: 0 0 0 0 [165360.368119] Node 1 Normal free:46848kB min:45728kB low:57160kB high:68592kB active_anon:32kB inactive_anon:224kB active_file:216kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:224kB mapped:4320kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no [165360.368121] lowmem_reserve[]: 0 0 0 0 [165360.368129] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165360.368136] Node 0 DMA32: 90*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165360.368143] Node 0 Normal: 253*4kB (UEM) 173*8kB (UEM) 208*16kB (UEM) 283*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37244kB [165360.368150] Node 1 Normal: 544*4kB (UEM) 365*8kB (UEM) 210*16kB (EM) 440*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47176kB [165360.368151] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165360.368152] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165360.368153] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165360.368154] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165360.368155] 1800 total pagecache pages [165360.368155] 86 pages in swap cache [165360.368156] Swap cache stats: add 308176, delete 308090, find 91759/147934 [165360.368157] Free swap = 16247876kB [165360.368157] Total swap = 16319484kB [165360.368158] 8369063 pages RAM [165360.368159] 0 pages HighMem/MovableOnly [165360.368159] 241261 pages reserved [165360.368316] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165360.368318] CPU: 0 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165360.368319] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165360.368322] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165360.368324] 00000000000080d0 00000000fa2ca73a ffff88073f6577c0 ffffffff816a3e2d [165360.368326] ffff88073f657850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165360.368327] ffffffffffffff00 000080d000000000 ffff88073f657820 00000000fa2ca73a [165360.368328] Call Trace: [165360.368330] [] dump_stack+0x19/0x1b [165360.368333] [] warn_alloc_failed+0x110/0x180 [165360.368334] [] ? drain_pages+0xb0/0xb0 [165360.368336] [] __alloc_pages_slowpath+0x6b6/0x724 [165360.368338] [] __alloc_pages_nodemask+0x405/0x420 [165360.368340] [] alloc_pages_current+0x98/0x110 [165360.368343] [] __get_free_pages+0xe/0x40 [165360.368345] [] swiotlb_alloc_coherent+0x5e/0x150 [165360.368347] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165360.368355] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165360.368362] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165360.368368] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165360.368370] [] ? dma_pool_free+0xa7/0xd0 [165360.368375] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165360.368380] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165360.368386] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165360.368393] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165360.368396] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165360.368400] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165360.368405] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165360.368408] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165360.368410] [] process_one_work+0x17a/0x440 [165360.368412] [] worker_thread+0x126/0x3c0 [165360.368414] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165360.368416] [] kthread+0xcf/0xe0 [165360.368417] [] ? insert_kthread_work+0x40/0x40 [165360.368419] [] ret_from_fork+0x58/0x90 [165360.368421] [] ? insert_kthread_work+0x40/0x40 [165360.368421] Mem-Info: [165360.368427] active_anon:8 inactive_anon:0 isolated_anon:0 active_file:20 inactive_file:46 isolated_file:0 unevictable:6774 dirty:0 writeback:122 unstable:0 slab_reclaimable:16107 slab_unreclaimable:564384 mapped:1543 shmem:0 pagetables:1560 bounce:0 free:39703 free_pcp:0 free_cma:0 [165360.368432] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165360.368434] lowmem_reserve[]: 0 2580 15620 15620 [165360.368439] Node 0 DMA32 free:59584kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165360.368441] lowmem_reserve[]: 0 0 13040 13040 [165360.368446] Node 0 Normal free:36800kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:292kB mapped:1508kB shmem:0kB slab_reclaimable:26712kB slab_unreclaimable:1101872kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165360.368448] lowmem_reserve[]: 0 0 0 0 [165360.368453] Node 1 Normal free:46848kB min:45728kB low:57160kB high:68592kB active_anon:32kB inactive_anon:224kB active_file:216kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:224kB mapped:4320kB shmem:0kB slab_reclaimable:33432kB slab_unreclaimable:958804kB kernel_stack:17376kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:65 all_unreclaimable? no [165360.368455] lowmem_reserve[]: 0 0 0 0 [165360.368462] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165360.368470] Node 0 DMA32: 90*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165360.368476] Node 0 Normal: 253*4kB (UEM) 173*8kB (UEM) 208*16kB (UEM) 283*32kB (UEM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37244kB [165360.368484] Node 1 Normal: 544*4kB (UEM) 365*8kB (UEM) 210*16kB (EM) 440*32kB (UM) 313*64kB (UEM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47176kB [165360.368485] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165360.368486] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165360.368486] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165360.368487] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165360.368488] 1800 total pagecache pages [165360.368488] 86 pages in swap cache [165360.368489] Swap cache stats: add 308176, delete 308090, find 91759/147934 [165360.368490] Free swap = 16247876kB [165360.368490] Total swap = 16319484kB [165360.368491] 8369063 pages RAM [165360.368491] 0 pages HighMem/MovableOnly [165360.368491] 241261 pages reserved [165362.302854] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165362.302856] CPU: 21 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165362.302856] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165362.302861] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165362.302862] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165362.302863] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165362.302864] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165362.302864] Call Trace: [165362.302867] [] dump_stack+0x19/0x1b [165362.302869] [] warn_alloc_failed+0x110/0x180 [165362.302870] [] __alloc_pages_slowpath+0x6b6/0x724 [165362.302872] [] __alloc_pages_nodemask+0x405/0x420 [165362.302874] [] dma_generic_alloc_coherent+0x8f/0x140 [165362.302875] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165362.302884] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165362.302889] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165362.302895] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165362.302897] [] ? dma_pool_free+0xa7/0xd0 [165362.302901] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165362.302905] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165362.302910] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165362.302916] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165362.302918] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165362.302922] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165362.302926] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165362.302928] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165362.302930] [] process_one_work+0x17a/0x440 [165362.302931] [] worker_thread+0x126/0x3c0 [165362.302933] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165362.302934] [] kthread+0xcf/0xe0 [165362.302935] [] ? insert_kthread_work+0x40/0x40 [165362.302936] [] ret_from_fork+0x58/0x90 [165362.302937] [] ? insert_kthread_work+0x40/0x40 [165362.302938] Mem-Info: [165362.302942] active_anon:35 inactive_anon:0 isolated_anon:0 active_file:127 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:168 unstable:0 slab_reclaimable:16115 slab_unreclaimable:564388 mapped:1589 shmem:0 pagetables:1560 bounce:0 free:39379 free_pcp:0 free_cma:0 [165362.302945] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165362.302946] lowmem_reserve[]: 0 2580 15620 15620 [165362.302950] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:0kB active_file:32kB inactive_file:64kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:12427 all_unreclaimable? no [165362.302951] lowmem_reserve[]: 0 0 13040 13040 [165362.302955] Node 0 Normal free:36660kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:292kB mapped:1508kB shmem:0kB slab_reclaimable:26712kB slab_unreclaimable:1101872kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:432 all_unreclaimable? yes [165362.302956] lowmem_reserve[]: 0 0 0 0 [165362.302959] Node 1 Normal free:45796kB min:45728kB low:57160kB high:68592kB active_anon:132kB inactive_anon:0kB active_file:460kB inactive_file:276kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:408kB mapped:4504kB shmem:0kB slab_reclaimable:33464kB slab_unreclaimable:958820kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:49 all_unreclaimable? no [165362.302960] lowmem_reserve[]: 0 0 0 0 [165362.302965] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165362.302970] Node 0 DMA32: 93*4kB (UEM) 2347*8kB (UEM) 1297*16kB (UEM) 465*32kB (UM) 73*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59580kB [165362.302974] Node 0 Normal: 219*4kB (UEM) 172*8kB (UEM) 208*16kB (UEM) 282*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37068kB [165362.302978] Node 1 Normal: 513*4kB (UEM) 357*8kB (UEM) 216*16kB (UEM) 443*32kB (UM) 312*64kB (UM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47116kB [165362.302979] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165362.302980] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165362.302980] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165362.302981] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165362.302981] 2005 total pagecache pages [165362.302982] 124 pages in swap cache [165362.302982] Swap cache stats: add 308938, delete 308814, find 91992/148336 [165362.302983] Free swap = 16247864kB [165362.302983] Total swap = 16319484kB [165362.302984] 8369063 pages RAM [165362.302984] 0 pages HighMem/MovableOnly [165362.302984] 241261 pages reserved [165362.303158] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165362.303159] CPU: 21 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165362.303160] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165362.303162] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165362.303164] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165362.303165] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165362.303166] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165362.303166] Call Trace: [165362.303168] [] dump_stack+0x19/0x1b [165362.303170] [] warn_alloc_failed+0x110/0x180 [165362.303171] [] ? drain_pages+0xb0/0xb0 [165362.303172] [] __alloc_pages_slowpath+0x6b6/0x724 [165362.303173] [] __alloc_pages_nodemask+0x405/0x420 [165362.303175] [] alloc_pages_current+0x98/0x110 [165362.303176] [] __get_free_pages+0xe/0x40 [165362.303178] [] swiotlb_alloc_coherent+0x5e/0x150 [165362.303179] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165362.303185] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165362.303190] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165362.303195] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165362.303197] [] ? dma_pool_free+0xa7/0xd0 [165362.303201] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165362.303204] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165362.303209] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165362.303215] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165362.303217] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165362.303220] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165362.303224] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165362.303226] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165362.303227] [] process_one_work+0x17a/0x440 [165362.303228] [] worker_thread+0x126/0x3c0 [165362.303230] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165362.303231] [] kthread+0xcf/0xe0 [165362.303232] [] ? insert_kthread_work+0x40/0x40 [165362.303233] [] ret_from_fork+0x58/0x90 [165362.303234] [] ? insert_kthread_work+0x40/0x40 [165362.303235] Mem-Info: [165362.303238] active_anon:35 inactive_anon:0 isolated_anon:0 active_file:127 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:168 unstable:0 slab_reclaimable:16115 slab_unreclaimable:564388 mapped:1589 shmem:0 pagetables:1560 bounce:0 free:39379 free_pcp:0 free_cma:0 [165362.303242] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165362.303243] lowmem_reserve[]: 0 2580 15620 15620 [165362.303246] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:0kB active_file:32kB inactive_file:64kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:236kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:12429 all_unreclaimable? yes [165362.303247] lowmem_reserve[]: 0 0 13040 13040 [165362.303251] Node 0 Normal free:36660kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:292kB mapped:1508kB shmem:0kB slab_reclaimable:26712kB slab_unreclaimable:1101872kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:442 all_unreclaimable? yes [165362.303252] lowmem_reserve[]: 0 0 0 0 [165362.303256] Node 1 Normal free:45796kB min:45728kB low:57160kB high:68592kB active_anon:132kB inactive_anon:0kB active_file:460kB inactive_file:276kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:408kB mapped:4504kB shmem:0kB slab_reclaimable:33464kB slab_unreclaimable:958820kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:173 all_unreclaimable? no [165362.303257] lowmem_reserve[]: 0 0 0 0 [165362.303262] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165362.303267] Node 0 DMA32: 93*4kB (UEM) 2347*8kB (UEM) 1297*16kB (UEM) 465*32kB (UM) 73*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59580kB [165362.303271] Node 0 Normal: 219*4kB (UEM) 172*8kB (UEM) 208*16kB (UEM) 282*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37068kB [165362.303275] Node 1 Normal: 513*4kB (UEM) 357*8kB (UEM) 216*16kB (UEM) 443*32kB (UM) 312*64kB (UM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47116kB [165362.303276] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165362.303276] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165362.303277] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165362.303277] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165362.303278] 2005 total pagecache pages [165362.303278] 124 pages in swap cache [165362.303279] Swap cache stats: add 308938, delete 308814, find 91992/148336 [165362.303279] Free swap = 16247864kB [165362.303279] Total swap = 16319484kB [165362.303280] 8369063 pages RAM [165362.303280] 0 pages HighMem/MovableOnly [165362.303280] 241261 pages reserved [165362.373053] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165362.373054] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165362.373055] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165362.373058] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165362.373060] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165362.373061] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165362.373062] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165362.373062] Call Trace: [165362.373064] [] dump_stack+0x19/0x1b [165362.373066] [] warn_alloc_failed+0x110/0x180 [165362.373067] [] __alloc_pages_slowpath+0x6b6/0x724 [165362.373069] [] __alloc_pages_nodemask+0x405/0x420 [165362.373071] [] dma_generic_alloc_coherent+0x8f/0x140 [165362.373072] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165362.373080] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165362.373085] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165362.373090] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165362.373092] [] ? dma_pool_free+0xa7/0xd0 [165362.373096] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165362.373100] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165362.373105] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165362.373110] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165362.373112] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165362.373116] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165362.373120] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165362.373122] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165362.373123] [] process_one_work+0x17a/0x440 [165362.373125] [] worker_thread+0x126/0x3c0 [165362.373126] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165362.373127] [] kthread+0xcf/0xe0 [165362.373128] [] ? insert_kthread_work+0x40/0x40 [165362.373130] [] ret_from_fork+0x58/0x90 [165362.373131] [] ? insert_kthread_work+0x40/0x40 [165362.373131] Mem-Info: [165362.373135] active_anon:3 inactive_anon:39 isolated_anon:0 active_file:40 inactive_file:22 isolated_file:0 unevictable:6774 dirty:0 writeback:196 unstable:0 slab_reclaimable:16131 slab_unreclaimable:564388 mapped:1541 shmem:0 pagetables:1561 bounce:0 free:39691 free_pcp:0 free_cma:0 [165362.373139] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165362.373140] lowmem_reserve[]: 0 2580 15620 15620 [165362.373144] Node 0 DMA32 free:59576kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:20kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165362.373145] lowmem_reserve[]: 0 0 13040 13040 [165362.373148] Node 0 Normal free:36876kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:308kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101872kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165362.373149] lowmem_reserve[]: 0 0 0 0 [165362.373153] Node 1 Normal free:46732kB min:45728kB low:57160kB high:68592kB active_anon:8kB inactive_anon:380kB active_file:200kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:484kB mapped:4316kB shmem:0kB slab_reclaimable:33496kB slab_unreclaimable:958820kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165362.373154] lowmem_reserve[]: 0 0 0 0 [165362.373159] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165362.373164] Node 0 DMA32: 91*4kB (UEM) 2348*8kB (UEM) 1296*16kB (UEM) 464*32kB (UM) 74*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59596kB [165362.373168] Node 0 Normal: 249*4kB (UEM) 172*8kB (UEM) 208*16kB (UEM) 281*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165362.373172] Node 1 Normal: 512*4kB (UEM) 358*8kB (UEM) 216*16kB (UEM) 438*32kB (UM) 312*64kB (UM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46960kB [165362.373173] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165362.373173] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165362.373174] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165362.373175] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165362.373175] 1840 total pagecache pages [165362.373176] 123 pages in swap cache [165362.373176] Swap cache stats: add 308945, delete 308822, find 91996/148340 [165362.373177] Free swap = 16247868kB [165362.373177] Total swap = 16319484kB [165362.373178] 8369063 pages RAM [165362.373178] 0 pages HighMem/MovableOnly [165362.373178] 241261 pages reserved [165362.377306] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165362.377308] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165362.377309] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165362.377312] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165362.377313] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165362.377314] ffff8807e1567850 ffffffff81188820 0000000000000000 ffff88043ffdb000 [165362.377315] 0000000000000008 00000000000080d0 ffff8807e1567850 00000000a36943d5 [165362.377315] Call Trace: [165362.377318] [] dump_stack+0x19/0x1b [165362.377320] [] warn_alloc_failed+0x110/0x180 [165362.377321] [] __alloc_pages_slowpath+0x6b6/0x724 [165362.377322] [] __alloc_pages_nodemask+0x405/0x420 [165362.377324] [] alloc_pages_current+0x98/0x110 [165362.377326] [] __get_free_pages+0xe/0x40 [165362.377327] [] swiotlb_alloc_coherent+0x5e/0x150 [165362.377329] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165362.377337] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165362.377342] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165362.377347] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165362.377349] [] ? dma_pool_free+0xa7/0xd0 [165362.377353] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165362.377357] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165362.377361] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165362.377367] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165362.377369] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165362.377373] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165362.377377] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165362.377379] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165362.377381] [] process_one_work+0x17a/0x440 [165362.377382] [] worker_thread+0x126/0x3c0 [165362.377383] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165362.377384] [] kthread+0xcf/0xe0 [165362.377385] [] ? insert_kthread_work+0x40/0x40 [165362.377387] [] ret_from_fork+0x58/0x90 [165362.377388] [] ? insert_kthread_work+0x40/0x40 [165362.377388] Mem-Info: [165362.377392] active_anon:3 inactive_anon:39 isolated_anon:0 active_file:40 inactive_file:22 isolated_file:0 unevictable:6774 dirty:0 writeback:196 unstable:0 slab_reclaimable:16131 slab_unreclaimable:564388 mapped:1541 shmem:0 pagetables:1561 bounce:0 free:39677 free_pcp:0 free_cma:0 [165362.377395] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165362.377397] lowmem_reserve[]: 0 2580 15620 15620 [165362.377400] Node 0 DMA32 free:59520kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:20kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165362.377401] lowmem_reserve[]: 0 0 13040 13040 [165362.377405] Node 0 Normal free:36876kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:308kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101872kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165362.377406] lowmem_reserve[]: 0 0 0 0 [165362.377410] Node 1 Normal free:46732kB min:45728kB low:57160kB high:68592kB active_anon:8kB inactive_anon:380kB active_file:200kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:484kB mapped:4316kB shmem:0kB slab_reclaimable:33496kB slab_unreclaimable:958820kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:212 all_unreclaimable? no [165362.377410] lowmem_reserve[]: 0 0 0 0 [165362.377416] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165362.377420] Node 0 DMA32: 92*4kB (UEM) 2350*8kB (UEM) 1296*16kB (UEM) 464*32kB (UM) 73*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59552kB [165362.377424] Node 0 Normal: 249*4kB (UEM) 172*8kB (UEM) 208*16kB (UEM) 281*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165362.377429] Node 1 Normal: 512*4kB (UEM) 358*8kB (UEM) 216*16kB (UEM) 438*32kB (UM) 312*64kB (UM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46960kB [165362.377429] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165362.377430] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165362.377430] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165362.377431] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165362.377431] 1840 total pagecache pages [165362.377432] 123 pages in swap cache [165362.377432] Swap cache stats: add 308945, delete 308822, find 91996/148340 [165362.377433] Free swap = 16247868kB [165362.377433] Total swap = 16319484kB [165362.377434] 8369063 pages RAM [165362.377434] 0 pages HighMem/MovableOnly [165362.377434] 241261 pages reserved [165367.367693] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165367.367695] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165367.367695] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165367.367699] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165367.367701] 00000000000080d0 00000000a36943d5 ffff8807e1567810 ffffffff816a3e2d [165367.367702] ffff8807e15678a0 ffffffff81188820 0000000000000000 00000000ffffffff [165367.367703] ffffffffffffff00 000080d000000000 ffff8807e1567870 00000000a36943d5 [165367.367703] Call Trace: [165367.367706] [] dump_stack+0x19/0x1b [165367.367708] [] warn_alloc_failed+0x110/0x180 [165367.367710] [] __alloc_pages_slowpath+0x6b6/0x724 [165367.367711] [] __alloc_pages_nodemask+0x405/0x420 [165367.367713] [] dma_generic_alloc_coherent+0x8f/0x140 [165367.367715] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165367.367723] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165367.367728] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165367.367734] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165367.367735] [] ? dma_pool_free+0xa7/0xd0 [165367.367740] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165367.367744] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165367.367749] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165367.367755] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165367.367757] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165367.367761] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165367.367765] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165367.367767] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165367.367769] [] process_one_work+0x17a/0x440 [165367.367770] [] worker_thread+0x126/0x3c0 [165367.367771] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165367.367772] [] kthread+0xcf/0xe0 [165367.367774] [] ? insert_kthread_work+0x40/0x40 [165367.367775] [] ret_from_fork+0x58/0x90 [165367.367776] [] ? insert_kthread_work+0x40/0x40 [165367.367776] Mem-Info: [165367.367780] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:41 inactive_file:22 isolated_file:0 unevictable:6774 dirty:0 writeback:40 unstable:0 slab_reclaimable:16131 slab_unreclaimable:564396 mapped:1541 shmem:0 pagetables:1561 bounce:0 free:39819 free_pcp:0 free_cma:0 [165367.367784] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165367.367785] lowmem_reserve[]: 0 2580 15620 15620 [165367.367789] Node 0 DMA32 free:59576kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:20kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no [165367.367790] lowmem_reserve[]: 0 0 13040 13040 [165367.367794] Node 0 Normal free:36988kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101904kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:782 all_unreclaimable? yes [165367.367795] lowmem_reserve[]: 0 0 0 0 [165367.367798] Node 1 Normal free:47132kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:4kB active_file:200kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:284kB mapped:4316kB shmem:0kB slab_reclaimable:33496kB slab_unreclaimable:958820kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165367.367799] lowmem_reserve[]: 0 0 0 0 [165367.367804] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165367.367809] Node 0 DMA32: 90*4kB (UEM) 2348*8kB (UEM) 1295*16kB (UEM) 463*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165367.367813] Node 0 Normal: 249*4kB (UEM) 171*8kB (UEM) 210*16kB (UEM) 281*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37180kB [165367.367817] Node 1 Normal: 521*4kB (UEM) 383*8kB (UEM) 216*16kB (UEM) 438*32kB (UM) 312*64kB (UM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47196kB [165367.367818] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165367.367818] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165367.367819] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165367.367819] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165367.367820] 1727 total pagecache pages [165367.367820] 59 pages in swap cache [165367.367821] Swap cache stats: add 309005, delete 308946, find 92018/148375 [165367.367821] Free swap = 16247864kB [165367.367822] Total swap = 16319484kB [165367.367822] 8369063 pages RAM [165367.367823] 0 pages HighMem/MovableOnly [165367.367823] 241261 pages reserved [165367.368026] kworker/u480:3: page allocation failure: order:8, mode:0x80d0 [165367.368027] CPU: 0 PID: 67624 Comm: kworker/u480:3 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165367.368028] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165367.368031] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165367.368032] 00000000000080d0 00000000a36943d5 ffff8807e15677c0 ffffffff816a3e2d [165367.368033] ffff8807e1567850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165367.368034] ffffffffffffff00 000080d000000000 ffff8807e1567820 00000000a36943d5 [165367.368034] Call Trace: [165367.368036] [] dump_stack+0x19/0x1b [165367.368037] [] warn_alloc_failed+0x110/0x180 [165367.368038] [] ? drain_pages+0xb0/0xb0 [165367.368040] [] __alloc_pages_slowpath+0x6b6/0x724 [165367.368041] [] __alloc_pages_nodemask+0x405/0x420 [165367.368042] [] alloc_pages_current+0x98/0x110 [165367.368044] [] __get_free_pages+0xe/0x40 [165367.368045] [] swiotlb_alloc_coherent+0x5e/0x150 [165367.368047] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165367.368053] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165367.368058] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165367.368062] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165367.368064] [] ? dma_pool_free+0xa7/0xd0 [165367.368068] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165367.368072] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165367.368076] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165367.368082] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165367.368084] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165367.368087] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165367.368091] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165367.368093] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165367.368094] [] process_one_work+0x17a/0x440 [165367.368096] [] worker_thread+0x126/0x3c0 [165367.368097] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165367.368098] [] kthread+0xcf/0xe0 [165367.368099] [] ? insert_kthread_work+0x40/0x40 [165367.368100] [] ret_from_fork+0x58/0x90 [165367.368101] [] ? insert_kthread_work+0x40/0x40 [165367.368102] Mem-Info: [165367.368106] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:41 inactive_file:22 isolated_file:0 unevictable:6774 dirty:0 writeback:40 unstable:0 slab_reclaimable:16131 slab_unreclaimable:564396 mapped:1541 shmem:0 pagetables:1561 bounce:0 free:39819 free_pcp:0 free_cma:0 [165367.368109] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165367.368110] lowmem_reserve[]: 0 2580 15620 15620 [165367.368114] Node 0 DMA32 free:59576kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:20kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no [165367.368114] lowmem_reserve[]: 0 0 13040 13040 [165367.368118] Node 0 Normal free:36988kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101904kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:787 all_unreclaimable? yes [165367.368119] lowmem_reserve[]: 0 0 0 0 [165367.368122] Node 1 Normal free:47132kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:4kB active_file:200kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:284kB mapped:4316kB shmem:0kB slab_reclaimable:33496kB slab_unreclaimable:958820kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165367.368123] lowmem_reserve[]: 0 0 0 0 [165367.368127] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165367.368132] Node 0 DMA32: 90*4kB (UEM) 2348*8kB (UEM) 1295*16kB (UEM) 463*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165367.368136] Node 0 Normal: 249*4kB (UEM) 171*8kB (UEM) 210*16kB (UEM) 281*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37180kB [165367.368140] Node 1 Normal: 521*4kB (UEM) 383*8kB (UEM) 216*16kB (UEM) 438*32kB (UM) 312*64kB (UM) 36*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47196kB [165367.368141] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165367.368141] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165367.368142] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165367.368142] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165367.368143] 1727 total pagecache pages [165367.368143] 59 pages in swap cache [165367.368144] Swap cache stats: add 309005, delete 308946, find 92018/148375 [165367.368144] Free swap = 16247864kB [165367.368144] Total swap = 16319484kB [165367.368145] 8369063 pages RAM [165367.368145] 0 pages HighMem/MovableOnly [165367.368145] 241261 pages reserved [165369.281386] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165369.281388] CPU: 24 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165369.281389] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165369.281397] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165369.281399] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165369.281400] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165369.281401] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165369.281401] Call Trace: [165369.281408] [] dump_stack+0x19/0x1b [165369.281413] [] warn_alloc_failed+0x110/0x180 [165369.281414] [] __alloc_pages_slowpath+0x6b6/0x724 [165369.281416] [] __alloc_pages_nodemask+0x405/0x420 [165369.281419] [] dma_generic_alloc_coherent+0x8f/0x140 [165369.281422] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165369.281444] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165369.281450] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165369.281457] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165369.281460] [] ? dma_pool_free+0xa7/0xd0 [165369.281465] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165369.281468] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165369.281474] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165369.281485] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165369.281487] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165369.281495] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165369.281500] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165369.281502] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165369.281504] [] process_one_work+0x17a/0x440 [165369.281506] [] worker_thread+0x126/0x3c0 [165369.281508] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165369.281509] [] kthread+0xcf/0xe0 [165369.281511] [] ? insert_kthread_work+0x40/0x40 [165369.281513] [] ret_from_fork+0x58/0x90 [165369.281515] [] ? insert_kthread_work+0x40/0x40 [165369.281515] Mem-Info: [165369.281520] active_anon:10 inactive_anon:18 isolated_anon:0 active_file:143 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16147 slab_unreclaimable:564396 mapped:1571 shmem:2 pagetables:1561 bounce:0 free:39470 free_pcp:3 free_cma:0 [165369.281525] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165369.281529] lowmem_reserve[]: 0 2580 15620 15620 [165369.281533] Node 0 DMA32 free:59512kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:196kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no [165369.281534] lowmem_reserve[]: 0 0 13040 13040 [165369.281538] Node 0 Normal free:36988kB min:37012kB low:46264kB high:55516kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:4kB slab_reclaimable:26744kB slab_unreclaimable:1101904kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:199 all_unreclaimable? yes [165369.281538] lowmem_reserve[]: 0 0 0 0 [165369.281542] Node 1 Normal free:45800kB min:45728kB low:57160kB high:68592kB active_anon:32kB inactive_anon:360kB active_file:428kB inactive_file:140kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4436kB shmem:4kB slab_reclaimable:33560kB slab_unreclaimable:958820kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:276 all_unreclaimable? no [165369.281543] lowmem_reserve[]: 0 0 0 0 [165369.281552] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165369.281556] Node 0 DMA32: 89*4kB (UEM) 2348*8kB (UEM) 1295*16kB (UEM) 463*32kB (UM) 75*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59604kB [165369.281561] Node 0 Normal: 248*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 281*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37200kB [165369.281565] Node 1 Normal: 478*4kB (UEM) 371*8kB (UEM) 216*16kB (UEM) 444*32kB (UEM) 312*64kB (UM) 35*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46992kB [165369.281566] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165369.281567] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165369.281568] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165369.281568] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165369.281569] 1950 total pagecache pages [165369.281569] 53 pages in swap cache [165369.281570] Swap cache stats: add 309627, delete 309574, find 92187/148674 [165369.281570] Free swap = 16247864kB [165369.281571] Total swap = 16319484kB [165369.281571] 8369063 pages RAM [165369.281572] 0 pages HighMem/MovableOnly [165369.281572] 241261 pages reserved [165369.286154] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165369.286156] CPU: 24 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165369.286157] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165369.286160] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165369.286162] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165369.286163] ffff880406203850 ffffffff81188820 0000000000000000 ffff88083ffd8000 [165369.286164] 0000000000000008 00000000000080d0 ffff880406203850 0000000039b5c568 [165369.286164] Call Trace: [165369.286167] [] dump_stack+0x19/0x1b [165369.286169] [] warn_alloc_failed+0x110/0x180 [165369.286171] [] __alloc_pages_slowpath+0x6b6/0x724 [165369.286172] [] __alloc_pages_nodemask+0x405/0x420 [165369.286175] [] alloc_pages_current+0x98/0x110 [165369.286178] [] __get_free_pages+0xe/0x40 [165369.286181] [] swiotlb_alloc_coherent+0x5e/0x150 [165369.286184] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165369.286193] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165369.286199] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165369.286204] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165369.286207] [] ? dma_pool_free+0xa7/0xd0 [165369.286212] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165369.286216] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165369.286221] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165369.286228] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165369.286230] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165369.286235] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165369.286240] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165369.286243] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165369.286245] [] process_one_work+0x17a/0x440 [165369.286247] [] worker_thread+0x126/0x3c0 [165369.286249] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165369.286250] [] kthread+0xcf/0xe0 [165369.286251] [] ? insert_kthread_work+0x40/0x40 [165369.286253] [] ret_from_fork+0x58/0x90 [165369.286254] [] ? insert_kthread_work+0x40/0x40 [165369.286254] Mem-Info: [165369.286258] active_anon:10 inactive_anon:8 isolated_anon:0 active_file:147 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16147 slab_unreclaimable:564396 mapped:1563 shmem:2 pagetables:1561 bounce:0 free:39518 free_pcp:0 free_cma:0 [165369.286262] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165369.286263] lowmem_reserve[]: 0 2580 15620 15620 [165369.286267] Node 0 DMA32 free:59460kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:196kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165369.286268] lowmem_reserve[]: 0 0 13040 13040 [165369.286271] Node 0 Normal free:36988kB min:37012kB low:46264kB high:55516kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:4kB slab_reclaimable:26744kB slab_unreclaimable:1101904kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:199 all_unreclaimable? yes [165369.286273] lowmem_reserve[]: 0 0 0 0 [165369.286278] Node 1 Normal free:46044kB min:45728kB low:57160kB high:68592kB active_anon:32kB inactive_anon:316kB active_file:460kB inactive_file:0kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4404kB shmem:4kB slab_reclaimable:33560kB slab_unreclaimable:958820kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? no [165369.286279] lowmem_reserve[]: 0 0 0 0 [165369.286285] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165369.286289] Node 0 DMA32: 88*4kB (UEM) 2347*8kB (UEM) 1294*16kB (UEM) 461*32kB (UM) 76*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59576kB [165369.286293] Node 0 Normal: 248*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 281*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37200kB [165369.286298] Node 1 Normal: 498*4kB (UEM) 371*8kB (UEM) 216*16kB (UEM) 443*32kB (UM) 312*64kB (UM) 35*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47040kB [165369.286298] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165369.286299] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165369.286300] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165369.286301] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165369.286301] 1901 total pagecache pages [165369.286302] 61 pages in swap cache [165369.286302] Swap cache stats: add 309635, delete 309574, find 92187/148674 [165369.286303] Free swap = 16247864kB [165369.286303] Total swap = 16319484kB [165369.286304] 8369063 pages RAM [165369.286304] 0 pages HighMem/MovableOnly [165369.286304] 241261 pages reserved [165369.365926] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165369.365928] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165369.365929] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165369.365934] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165369.365936] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165369.365938] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165369.365939] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165369.365940] Call Trace: [165369.365943] [] dump_stack+0x19/0x1b [165369.365946] [] warn_alloc_failed+0x110/0x180 [165369.365947] [] __alloc_pages_slowpath+0x6b6/0x724 [165369.365950] [] __alloc_pages_nodemask+0x405/0x420 [165369.365952] [] dma_generic_alloc_coherent+0x8f/0x140 [165369.365954] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165369.365967] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165369.365973] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165369.365980] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165369.365982] [] ? dma_pool_free+0xa7/0xd0 [165369.365988] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165369.365993] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165369.365999] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165369.366008] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165369.366011] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165369.366017] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165369.366022] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165369.366026] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165369.366027] [] process_one_work+0x17a/0x440 [165369.366029] [] worker_thread+0x126/0x3c0 [165369.366031] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165369.366033] [] kthread+0xcf/0xe0 [165369.366034] [] ? insert_kthread_work+0x40/0x40 [165369.366036] [] ret_from_fork+0x58/0x90 [165369.366037] [] ? insert_kthread_work+0x40/0x40 [165369.366038] Mem-Info: [165369.366044] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:19 inactive_file:45 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16163 slab_unreclaimable:564384 mapped:1542 shmem:0 pagetables:1561 bounce:0 free:39802 free_pcp:0 free_cma:0 [165369.366050] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165369.366052] lowmem_reserve[]: 0 2580 15620 15620 [165369.366058] Node 0 DMA32 free:59616kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:4kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:603 all_unreclaimable? yes [165369.366059] lowmem_reserve[]: 0 0 13040 13040 [165369.366065] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101904kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165369.366066] lowmem_reserve[]: 0 0 0 0 [165369.366072] Node 1 Normal free:47004kB min:45728kB low:57160kB high:68592kB active_anon:4kB inactive_anon:228kB active_file:204kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:33624kB slab_unreclaimable:958772kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165369.366073] lowmem_reserve[]: 0 0 0 0 [165369.366081] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165369.366089] Node 0 DMA32: 88*4kB (UEM) 2346*8kB (UEM) 1293*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59616kB [165369.366096] Node 0 Normal: 246*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 281*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37192kB [165369.366102] Node 1 Normal: 498*4kB (UEM) 371*8kB (UEM) 217*16kB (UEM) 444*32kB (UM) 312*64kB (UM) 35*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47088kB [165369.366103] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165369.366104] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165369.366105] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165369.366106] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165369.366106] 1781 total pagecache pages [165369.366107] 61 pages in swap cache [165369.366108] Swap cache stats: add 309635, delete 309574, find 92187/148674 [165369.366109] Free swap = 16247864kB [165369.366109] Total swap = 16319484kB [165369.366110] 8369063 pages RAM [165369.366111] 0 pages HighMem/MovableOnly [165369.366111] 241261 pages reserved [165369.366341] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165369.366343] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165369.366344] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165369.366348] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165369.366349] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165369.366351] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165369.366352] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165369.366353] Call Trace: [165369.366355] [] dump_stack+0x19/0x1b [165369.366358] [] warn_alloc_failed+0x110/0x180 [165369.366359] [] ? drain_pages+0xb0/0xb0 [165369.366361] [] __alloc_pages_slowpath+0x6b6/0x724 [165369.366363] [] __alloc_pages_nodemask+0x405/0x420 [165369.366365] [] alloc_pages_current+0x98/0x110 [165369.366367] [] __get_free_pages+0xe/0x40 [165369.366369] [] swiotlb_alloc_coherent+0x5e/0x150 [165369.366372] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165369.366380] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165369.366386] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165369.366392] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165369.366394] [] ? dma_pool_free+0xa7/0xd0 [165369.366400] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165369.366405] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165369.366410] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165369.366417] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165369.366420] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165369.366425] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165369.366430] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165369.366433] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165369.366435] [] process_one_work+0x17a/0x440 [165369.366436] [] worker_thread+0x126/0x3c0 [165369.366438] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165369.366440] [] kthread+0xcf/0xe0 [165369.366442] [] ? insert_kthread_work+0x40/0x40 [165369.366443] [] ret_from_fork+0x58/0x90 [165369.366445] [] ? insert_kthread_work+0x40/0x40 [165369.366446] Mem-Info: [165369.366451] active_anon:1 inactive_anon:0 isolated_anon:0 active_file:19 inactive_file:45 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16163 slab_unreclaimable:564384 mapped:1542 shmem:0 pagetables:1561 bounce:0 free:39802 free_pcp:0 free_cma:0 [165369.366456] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165369.366458] lowmem_reserve[]: 0 2580 15620 15620 [165369.366463] Node 0 DMA32 free:59616kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:4kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:603 all_unreclaimable? yes [165369.366465] lowmem_reserve[]: 0 0 13040 13040 [165369.366470] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101904kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165369.366471] lowmem_reserve[]: 0 0 0 0 [165369.366476] Node 1 Normal free:47004kB min:45728kB low:57160kB high:68592kB active_anon:4kB inactive_anon:228kB active_file:204kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:33624kB slab_unreclaimable:958772kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165369.366477] lowmem_reserve[]: 0 0 0 0 [165369.366485] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165369.366492] Node 0 DMA32: 88*4kB (UEM) 2346*8kB (UEM) 1293*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59616kB [165369.366498] Node 0 Normal: 246*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 281*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37192kB [165369.366505] Node 1 Normal: 498*4kB (UEM) 371*8kB (UEM) 217*16kB (UEM) 444*32kB (UM) 312*64kB (UM) 35*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47088kB [165369.366507] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165369.366508] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165369.366509] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165369.366510] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165369.366510] 1781 total pagecache pages [165369.366511] 61 pages in swap cache [165369.366512] Swap cache stats: add 309635, delete 309574, find 92187/148674 [165369.366512] Free swap = 16247864kB [165369.366513] Total swap = 16319484kB [165369.366513] 8369063 pages RAM [165369.366514] 0 pages HighMem/MovableOnly [165369.366514] 241261 pages reserved [165371.319692] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165371.319693] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165371.319694] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165371.319697] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165371.319699] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165371.319700] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165371.319701] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165371.319701] Call Trace: [165371.319704] [] dump_stack+0x19/0x1b [165371.319706] [] warn_alloc_failed+0x110/0x180 [165371.319707] [] __alloc_pages_slowpath+0x6b6/0x724 [165371.319708] [] __alloc_pages_nodemask+0x405/0x420 [165371.319710] [] dma_generic_alloc_coherent+0x8f/0x140 [165371.319711] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165371.319720] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165371.319725] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165371.319730] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165371.319731] [] ? dma_pool_free+0xa7/0xd0 [165371.319736] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165371.319740] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165371.319744] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165371.319750] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165371.319752] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165371.319756] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165371.319759] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165371.319762] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165371.319763] [] process_one_work+0x17a/0x440 [165371.319764] [] worker_thread+0x126/0x3c0 [165371.319766] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165371.319767] [] kthread+0xcf/0xe0 [165371.319768] [] ? insert_kthread_work+0x40/0x40 [165371.319769] [] ret_from_fork+0x58/0x90 [165371.319770] [] ? insert_kthread_work+0x40/0x40 [165371.319771] Mem-Info: [165371.319775] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:62 inactive_file:97 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16163 slab_unreclaimable:564392 mapped:1599 shmem:0 pagetables:1561 bounce:0 free:39498 free_pcp:28 free_cma:0 [165371.319778] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165371.319780] lowmem_reserve[]: 0 2580 15620 15620 [165371.319783] Node 0 DMA32 free:59548kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:52kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:4kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165371.319784] lowmem_reserve[]: 0 0 13040 13040 [165371.319788] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:154 all_unreclaimable? yes [165371.319788] lowmem_reserve[]: 0 0 0 0 [165371.319792] Node 1 Normal free:45900kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:92kB active_file:344kB inactive_file:608kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4548kB shmem:0kB slab_reclaimable:33624kB slab_unreclaimable:958772kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165371.319793] lowmem_reserve[]: 0 0 0 0 [165371.319798] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165371.319802] Node 0 DMA32: 89*4kB (UEM) 2347*8kB (UEM) 1290*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59580kB [165371.319807] Node 0 Normal: 245*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165371.319811] Node 1 Normal: 418*4kB (UEM) 366*8kB (UEM) 217*16kB (UEM) 445*32kB (UEM) 313*64kB (UEM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46696kB [165371.319811] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165371.319812] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165371.319813] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165371.319813] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165371.319814] 1927 total pagecache pages [165371.319814] 64 pages in swap cache [165371.319815] Swap cache stats: add 310346, delete 310282, find 92407/149055 [165371.319815] Free swap = 16247868kB [165371.319815] Total swap = 16319484kB [165371.319816] 8369063 pages RAM [165371.319816] 0 pages HighMem/MovableOnly [165371.319817] 241261 pages reserved [165371.319984] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165371.319985] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165371.319986] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165371.319988] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165371.319989] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165371.319990] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165371.319991] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165371.319992] Call Trace: [165371.319994] [] dump_stack+0x19/0x1b [165371.319995] [] warn_alloc_failed+0x110/0x180 [165371.319996] [] ? drain_pages+0xb0/0xb0 [165371.319998] [] __alloc_pages_slowpath+0x6b6/0x724 [165371.319999] [] __alloc_pages_nodemask+0x405/0x420 [165371.320000] [] alloc_pages_current+0x98/0x110 [165371.320002] [] __get_free_pages+0xe/0x40 [165371.320004] [] swiotlb_alloc_coherent+0x5e/0x150 [165371.320005] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165371.320011] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165371.320016] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165371.320021] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165371.320022] [] ? dma_pool_free+0xa7/0xd0 [165371.320026] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165371.320030] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165371.320034] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165371.320040] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165371.320042] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165371.320045] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165371.320049] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165371.320051] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165371.320052] [] process_one_work+0x17a/0x440 [165371.320054] [] worker_thread+0x126/0x3c0 [165371.320055] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165371.320056] [] kthread+0xcf/0xe0 [165371.320057] [] ? insert_kthread_work+0x40/0x40 [165371.320058] [] ret_from_fork+0x58/0x90 [165371.320059] [] ? insert_kthread_work+0x40/0x40 [165371.320060] Mem-Info: [165371.320063] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:62 inactive_file:97 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16163 slab_unreclaimable:564392 mapped:1599 shmem:0 pagetables:1561 bounce:0 free:39498 free_pcp:0 free_cma:0 [165371.320067] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165371.320068] lowmem_reserve[]: 0 2580 15620 15620 [165371.320072] Node 0 DMA32 free:59548kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:52kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:4kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165371.320073] lowmem_reserve[]: 0 0 13040 13040 [165371.320076] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:154 all_unreclaimable? yes [165371.320077] lowmem_reserve[]: 0 0 0 0 [165371.320080] Node 1 Normal free:45900kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:92kB active_file:344kB inactive_file:608kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4548kB shmem:0kB slab_reclaimable:33624kB slab_unreclaimable:958772kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165371.320081] lowmem_reserve[]: 0 0 0 0 [165371.320086] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165371.320091] Node 0 DMA32: 89*4kB (UEM) 2347*8kB (UEM) 1290*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59580kB [165371.320095] Node 0 Normal: 245*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165371.320099] Node 1 Normal: 446*4kB (UEM) 366*8kB (UEM) 217*16kB (UEM) 445*32kB (UEM) 313*64kB (UEM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46808kB [165371.320100] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165371.320100] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165371.320101] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165371.320101] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165371.320102] 1927 total pagecache pages [165371.320102] 64 pages in swap cache [165371.320103] Swap cache stats: add 310346, delete 310282, find 92407/149055 [165371.320103] Free swap = 16247868kB [165371.320103] Total swap = 16319484kB [165371.320104] 8369063 pages RAM [165371.320104] 0 pages HighMem/MovableOnly [165371.320104] 241261 pages reserved [165373.112685] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165373.112687] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165373.112687] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165373.112691] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165373.112693] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165373.112694] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165373.112695] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165373.112695] Call Trace: [165373.112697] [] dump_stack+0x19/0x1b [165373.112699] [] warn_alloc_failed+0x110/0x180 [165373.112701] [] __alloc_pages_slowpath+0x6b6/0x724 [165373.112702] [] __alloc_pages_nodemask+0x405/0x420 [165373.112704] [] dma_generic_alloc_coherent+0x8f/0x140 [165373.112705] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165373.112713] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165373.112718] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165373.112724] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165373.112728] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165373.112732] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165373.112737] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165373.112742] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165373.112745] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165373.112749] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165373.112752] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165373.112754] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165373.112756] [] process_one_work+0x17a/0x440 [165373.112757] [] worker_thread+0x126/0x3c0 [165373.112759] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165373.112760] [] kthread+0xcf/0xe0 [165373.112761] [] ? insert_kthread_work+0x40/0x40 [165373.112762] [] ret_from_fork+0x58/0x90 [165373.112763] [] ? insert_kthread_work+0x40/0x40 [165373.112764] Mem-Info: [165373.112768] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:25 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16179 slab_unreclaimable:564392 mapped:1548 shmem:0 pagetables:1561 bounce:0 free:39529 free_pcp:0 free_cma:0 [165373.112772] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165373.112773] lowmem_reserve[]: 0 2580 15620 15620 [165373.112776] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:130 all_unreclaimable? yes [165373.112777] lowmem_reserve[]: 0 0 13040 13040 [165373.112781] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:154 all_unreclaimable? yes [165373.112781] lowmem_reserve[]: 0 0 0 0 [165373.112785] Node 1 Normal free:46136kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:228kB inactive_file:36kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4344kB shmem:0kB slab_reclaimable:33688kB slab_unreclaimable:958772kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165373.112786] lowmem_reserve[]: 0 0 0 0 [165373.112791] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165373.112795] Node 0 DMA32: 63*4kB (UEM) 2348*8kB (UEM) 1290*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59484kB [165373.112800] Node 0 Normal: 245*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165373.112804] Node 1 Normal: 562*4kB (UEM) 367*8kB (UEM) 218*16kB (UEM) 444*32kB (UEM) 312*64kB (UM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47200kB [165373.112805] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165373.112805] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165373.112806] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165373.112806] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165373.112807] 1824 total pagecache pages [165373.112807] 36 pages in swap cache [165373.112808] Swap cache stats: add 310620, delete 310584, find 92496/149207 [165373.112809] Free swap = 16247860kB [165373.112809] Total swap = 16319484kB [165373.112810] 8369063 pages RAM [165373.112810] 0 pages HighMem/MovableOnly [165373.112810] 241261 pages reserved [165373.113446] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165373.113447] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165373.113448] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165373.113451] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165373.113452] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165373.113453] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165373.113454] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165373.113454] Call Trace: [165373.113456] [] dump_stack+0x19/0x1b [165373.113458] [] warn_alloc_failed+0x110/0x180 [165373.113459] [] ? drain_pages+0xb0/0xb0 [165373.113460] [] __alloc_pages_slowpath+0x6b6/0x724 [165373.113461] [] __alloc_pages_nodemask+0x405/0x420 [165373.113463] [] alloc_pages_current+0x98/0x110 [165373.113464] [] __get_free_pages+0xe/0x40 [165373.113466] [] swiotlb_alloc_coherent+0x5e/0x150 [165373.113467] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165373.113474] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165373.113479] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165373.113483] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165373.113487] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165373.113491] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165373.113496] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165373.113501] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165373.113503] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165373.113506] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165373.113510] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165373.113512] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165373.113514] [] process_one_work+0x17a/0x440 [165373.113515] [] worker_thread+0x126/0x3c0 [165373.113516] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165373.113517] [] kthread+0xcf/0xe0 [165373.113518] [] ? insert_kthread_work+0x40/0x40 [165373.113520] [] ret_from_fork+0x58/0x90 [165373.113521] [] ? insert_kthread_work+0x40/0x40 [165373.113521] Mem-Info: [165373.113525] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:25 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16179 slab_unreclaimable:564392 mapped:1548 shmem:0 pagetables:1561 bounce:0 free:39529 free_pcp:0 free_cma:0 [165373.113528] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165373.113529] lowmem_reserve[]: 0 2580 15620 15620 [165373.113533] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:12kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:130 all_unreclaimable? yes [165373.113534] lowmem_reserve[]: 0 0 13040 13040 [165373.113537] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:154 all_unreclaimable? yes [165373.113538] lowmem_reserve[]: 0 0 0 0 [165373.113542] Node 1 Normal free:46136kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:228kB inactive_file:36kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4344kB shmem:0kB slab_reclaimable:33688kB slab_unreclaimable:958772kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165373.113542] lowmem_reserve[]: 0 0 0 0 [165373.113547] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165373.113552] Node 0 DMA32: 63*4kB (UEM) 2348*8kB (UEM) 1290*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59484kB [165373.113556] Node 0 Normal: 245*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165373.113560] Node 1 Normal: 550*4kB (UEM) 367*8kB (UEM) 218*16kB (UEM) 444*32kB (UEM) 312*64kB (UM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47152kB [165373.113561] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165373.113561] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165373.113562] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165373.113562] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165373.113563] 1824 total pagecache pages [165373.113563] 46 pages in swap cache [165373.113564] Swap cache stats: add 310630, delete 310584, find 92496/149207 [165373.113564] Free swap = 16247860kB [165373.113564] Total swap = 16319484kB [165373.113565] 8369063 pages RAM [165373.113565] 0 pages HighMem/MovableOnly [165373.113565] 241261 pages reserved [165374.373436] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165374.373438] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165374.373438] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165374.373442] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165374.373443] 00000000000080d0 00000000c340495d ffff8807e687b810 ffffffff816a3e2d [165374.373444] ffff8807e687b8a0 ffffffff81188820 0000000000000000 00000000ffffffff [165374.373445] ffffffffffffff00 000080d000000000 ffff8807e687b870 00000000c340495d [165374.373446] Call Trace: [165374.373448] [] dump_stack+0x19/0x1b [165374.373450] [] warn_alloc_failed+0x110/0x180 [165374.373451] [] __alloc_pages_slowpath+0x6b6/0x724 [165374.373453] [] __alloc_pages_nodemask+0x405/0x420 [165374.373454] [] dma_generic_alloc_coherent+0x8f/0x140 [165374.373456] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165374.373464] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165374.373469] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165374.373474] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165374.373476] [] ? dma_pool_free+0xa7/0xd0 [165374.373480] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165374.373484] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165374.373489] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165374.373494] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165374.373496] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165374.373500] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165374.373504] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165374.373506] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165374.373508] [] process_one_work+0x17a/0x440 [165374.373509] [] worker_thread+0x126/0x3c0 [165374.373510] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165374.373511] [] kthread+0xcf/0xe0 [165374.373513] [] ? insert_kthread_work+0x40/0x40 [165374.373514] [] ret_from_fork+0x58/0x90 [165374.373515] [] ? insert_kthread_work+0x40/0x40 [165374.373515] Mem-Info: [165374.373519] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:19 inactive_file:45 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16188 slab_unreclaimable:564392 mapped:1542 shmem:0 pagetables:1561 bounce:0 free:39832 free_pcp:0 free_cma:0 [165374.373523] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165374.373524] lowmem_reserve[]: 0 2580 15620 15620 [165374.373528] Node 0 DMA32 free:59600kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5 all_unreclaimable? no [165374.373529] lowmem_reserve[]: 0 0 13040 13040 [165374.373532] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:154 all_unreclaimable? yes [165374.373533] lowmem_reserve[]: 0 0 0 0 [165374.373536] Node 1 Normal free:47184kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:4kB active_file:204kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:33724kB slab_unreclaimable:958772kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165374.373537] lowmem_reserve[]: 0 0 0 0 [165374.373542] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165374.373547] Node 0 DMA32: 90*4kB (UEM) 2346*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165374.373551] Node 0 Normal: 245*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165374.373556] Node 1 Normal: 525*4kB (UEM) 375*8kB (UEM) 218*16kB (UEM) 440*32kB (UEM) 312*64kB (UM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46988kB [165374.373556] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165374.373557] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165374.373558] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165374.373558] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165374.373558] 1723 total pagecache pages [165374.373559] 63 pages in swap cache [165374.373560] Swap cache stats: add 310842, delete 310779, find 92558/149315 [165374.373560] Free swap = 16247860kB [165374.373560] Total swap = 16319484kB [165374.373561] 8369063 pages RAM [165374.373561] 0 pages HighMem/MovableOnly [165374.373562] 241261 pages reserved [165374.373865] kworker/u480:0: page allocation failure: order:8, mode:0x80d0 [165374.373866] CPU: 0 PID: 67623 Comm: kworker/u480:0 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165374.373867] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165374.373870] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165374.373871] 00000000000080d0 00000000c340495d ffff8807e687b7c0 ffffffff816a3e2d [165374.373872] ffff8807e687b850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165374.373873] ffffffffffffff00 000080d000000000 ffff8807e687b820 00000000c340495d [165374.373873] Call Trace: [165374.373875] [] dump_stack+0x19/0x1b [165374.373877] [] warn_alloc_failed+0x110/0x180 [165374.373878] [] ? drain_pages+0xb0/0xb0 [165374.373879] [] __alloc_pages_slowpath+0x6b6/0x724 [165374.373880] [] __alloc_pages_nodemask+0x405/0x420 [165374.373881] [] alloc_pages_current+0x98/0x110 [165374.373883] [] __get_free_pages+0xe/0x40 [165374.373885] [] swiotlb_alloc_coherent+0x5e/0x150 [165374.373886] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165374.373893] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165374.373898] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165374.373902] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165374.373904] [] ? dma_pool_free+0xa7/0xd0 [165374.373908] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165374.373912] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165374.373916] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165374.373922] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165374.373924] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165374.373927] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165374.373931] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165374.373933] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165374.373934] [] process_one_work+0x17a/0x440 [165374.373935] [] worker_thread+0x126/0x3c0 [165374.373937] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165374.373938] [] kthread+0xcf/0xe0 [165374.373939] [] ? insert_kthread_work+0x40/0x40 [165374.373940] [] ret_from_fork+0x58/0x90 [165374.373941] [] ? insert_kthread_work+0x40/0x40 [165374.373942] Mem-Info: [165374.373945] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:19 inactive_file:45 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16188 slab_unreclaimable:564392 mapped:1542 shmem:0 pagetables:1561 bounce:0 free:39832 free_pcp:0 free_cma:0 [165374.373949] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165374.373950] lowmem_reserve[]: 0 2580 15620 15620 [165374.373953] Node 0 DMA32 free:59600kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:20kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5 all_unreclaimable? no [165374.373954] lowmem_reserve[]: 0 0 13040 13040 [165374.373958] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:154 all_unreclaimable? yes [165374.373959] lowmem_reserve[]: 0 0 0 0 [165374.373962] Node 1 Normal free:47184kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:4kB active_file:204kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:33724kB slab_unreclaimable:958772kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165374.373963] lowmem_reserve[]: 0 0 0 0 [165374.373968] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165374.373973] Node 0 DMA32: 90*4kB (UEM) 2346*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59608kB [165374.373977] Node 0 Normal: 245*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165374.373981] Node 1 Normal: 525*4kB (UEM) 375*8kB (UEM) 218*16kB (UEM) 440*32kB (UEM) 312*64kB (UM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46988kB [165374.373982] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165374.373982] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165374.373983] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165374.373983] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165374.373984] 1723 total pagecache pages [165374.373984] 63 pages in swap cache [165374.373985] Swap cache stats: add 310842, delete 310779, find 92558/149315 [165374.373985] Free swap = 16247860kB [165374.373985] Total swap = 16319484kB [165374.373986] 8369063 pages RAM [165374.373986] 0 pages HighMem/MovableOnly [165374.373986] 241261 pages reserved [165376.315002] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165376.315004] CPU: 3 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165376.315005] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165376.315013] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165376.315015] 00000000000080d0 00000000fa2ca73a ffff88073f657810 ffffffff816a3e2d [165376.315016] ffff88073f6578a0 ffffffff81188820 0000000000000000 00000000ffffffff [165376.315017] ffffffffffffff00 000080d000000000 ffff88073f657870 00000000fa2ca73a [165376.315017] Call Trace: [165376.315023] [] dump_stack+0x19/0x1b [165376.315027] [] warn_alloc_failed+0x110/0x180 [165376.315028] [] __alloc_pages_slowpath+0x6b6/0x724 [165376.315030] [] __alloc_pages_nodemask+0x405/0x420 [165376.315033] [] dma_generic_alloc_coherent+0x8f/0x140 [165376.315035] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165376.315053] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165376.315058] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165376.315064] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165376.315067] [] ? dma_pool_free+0xa7/0xd0 [165376.315072] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165376.315076] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165376.315081] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165376.315088] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165376.315090] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165376.315096] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165376.315101] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165376.315103] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165376.315105] [] process_one_work+0x17a/0x440 [165376.315106] [] worker_thread+0x126/0x3c0 [165376.315108] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165376.315109] [] kthread+0xcf/0xe0 [165376.315110] [] ? insert_kthread_work+0x40/0x40 [165376.315112] [] ret_from_fork+0x58/0x90 [165376.315113] [] ? insert_kthread_work+0x40/0x40 [165376.315114] Mem-Info: [165376.315118] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:24 inactive_file:163 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16196 slab_unreclaimable:564396 mapped:1566 shmem:0 pagetables:1561 bounce:0 free:39469 free_pcp:0 free_cma:0 [165376.315123] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165376.315124] lowmem_reserve[]: 0 2580 15620 15620 [165376.315128] Node 0 DMA32 free:59520kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:24 all_unreclaimable? no [165376.315129] lowmem_reserve[]: 0 0 13040 13040 [165376.315132] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:154 all_unreclaimable? yes [165376.315133] lowmem_reserve[]: 0 0 0 0 [165376.315137] Node 1 Normal free:45812kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:188kB active_file:248kB inactive_file:852kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4416kB shmem:0kB slab_reclaimable:33756kB slab_unreclaimable:958788kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:648 all_unreclaimable? no [165376.315137] lowmem_reserve[]: 0 0 0 0 [165376.315143] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165376.315147] Node 0 DMA32: 89*4kB (UEM) 2346*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59604kB [165376.315152] Node 0 Normal: 245*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165376.315156] Node 1 Normal: 534*4kB (UEM) 352*8kB (UEM) 216*16kB (UEM) 444*32kB (UM) 312*64kB (UM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46936kB [165376.315157] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165376.315158] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165376.315158] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165376.315159] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165376.315159] 1908 total pagecache pages [165376.315160] 31 pages in swap cache [165376.315161] Swap cache stats: add 311511, delete 311480, find 92791/149698 [165376.315161] Free swap = 16247860kB [165376.315161] Total swap = 16319484kB [165376.315162] 8369063 pages RAM [165376.315162] 0 pages HighMem/MovableOnly [165376.315163] 241261 pages reserved [165376.315309] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165376.315310] CPU: 3 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165376.315311] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165376.315313] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165376.315314] 00000000000080d0 00000000fa2ca73a ffff88073f6577c0 ffffffff816a3e2d [165376.315315] ffff88073f657850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165376.315316] ffffffffffffff00 000080d000000000 ffff88073f657820 00000000fa2ca73a [165376.315317] Call Trace: [165376.315319] [] dump_stack+0x19/0x1b [165376.315320] [] warn_alloc_failed+0x110/0x180 [165376.315321] [] ? drain_pages+0xb0/0xb0 [165376.315323] [] __alloc_pages_slowpath+0x6b6/0x724 [165376.315324] [] __alloc_pages_nodemask+0x405/0x420 [165376.315326] [] alloc_pages_current+0x98/0x110 [165376.315328] [] __get_free_pages+0xe/0x40 [165376.315330] [] swiotlb_alloc_coherent+0x5e/0x150 [165376.315332] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165376.315338] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165376.315343] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165376.315347] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165376.315349] [] ? dma_pool_free+0xa7/0xd0 [165376.315353] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165376.315357] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165376.315361] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165376.315367] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165376.315369] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165376.315372] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165376.315376] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165376.315378] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165376.315379] [] process_one_work+0x17a/0x440 [165376.315380] [] worker_thread+0x126/0x3c0 [165376.315381] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165376.315383] [] kthread+0xcf/0xe0 [165376.315384] [] ? insert_kthread_work+0x40/0x40 [165376.315385] [] ret_from_fork+0x58/0x90 [165376.315386] [] ? insert_kthread_work+0x40/0x40 [165376.315386] Mem-Info: [165376.315390] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:24 inactive_file:163 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16196 slab_unreclaimable:564396 mapped:1566 shmem:0 pagetables:1561 bounce:0 free:39469 free_pcp:30 free_cma:0 [165376.315394] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165376.315395] lowmem_reserve[]: 0 2580 15620 15620 [165376.315399] Node 0 DMA32 free:59520kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:340kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:36 all_unreclaimable? no [165376.315400] lowmem_reserve[]: 0 0 13040 13040 [165376.315403] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:154 all_unreclaimable? yes [165376.315404] lowmem_reserve[]: 0 0 0 0 [165376.315407] Node 1 Normal free:45812kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:188kB active_file:248kB inactive_file:852kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4416kB shmem:0kB slab_reclaimable:33756kB slab_unreclaimable:958788kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165376.315408] lowmem_reserve[]: 0 0 0 0 [165376.315413] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165376.315418] Node 0 DMA32: 89*4kB (UEM) 2346*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59604kB [165376.315422] Node 0 Normal: 245*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37156kB [165376.315426] Node 1 Normal: 498*4kB (UEM) 352*8kB (UEM) 216*16kB (UEM) 444*32kB (UM) 312*64kB (UM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46792kB [165376.315427] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165376.315428] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165376.315428] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165376.315429] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165376.315429] 1908 total pagecache pages [165376.315430] 37 pages in swap cache [165376.315430] Swap cache stats: add 311517, delete 311480, find 92791/149698 [165376.315431] Free swap = 16247860kB [165376.315431] Total swap = 16319484kB [165376.315432] 8369063 pages RAM [165376.315432] 0 pages HighMem/MovableOnly [165376.315432] 241261 pages reserved [165377.305164] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165377.305165] CPU: 0 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165377.305166] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165377.305171] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165377.305173] 00000000000080d0 00000000fa2ca73a ffff88073f657810 ffffffff816a3e2d [165377.305174] ffff88073f6578a0 ffffffff81188820 0000000000000000 00000000ffffffff [165377.305175] ffffffffffffff00 000080d000000000 ffff88073f657870 00000000fa2ca73a [165377.305175] Call Trace: [165377.305178] [] dump_stack+0x19/0x1b [165377.305181] [] warn_alloc_failed+0x110/0x180 [165377.305182] [] __alloc_pages_slowpath+0x6b6/0x724 [165377.305184] [] __alloc_pages_nodemask+0x405/0x420 [165377.305186] [] dma_generic_alloc_coherent+0x8f/0x140 [165377.305188] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165377.305200] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165377.305205] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165377.305211] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165377.305215] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165377.305219] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165377.305224] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165377.305231] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165377.305233] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165377.305238] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165377.305243] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165377.305245] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165377.305252] [] process_one_work+0x17a/0x440 [165377.305254] [] worker_thread+0x126/0x3c0 [165377.305255] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165377.305256] [] kthread+0xcf/0xe0 [165377.305258] [] ? insert_kthread_work+0x40/0x40 [165377.305259] [] ret_from_fork+0x58/0x90 [165377.305260] [] ? insert_kthread_work+0x40/0x40 [165377.305261] Mem-Info: [165377.305265] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:89 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16196 slab_unreclaimable:564396 mapped:1543 shmem:0 pagetables:1561 bounce:0 free:39848 free_pcp:0 free_cma:0 [165377.305269] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165377.305270] lowmem_reserve[]: 0 2580 15620 15620 [165377.305274] Node 0 DMA32 free:59572kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:12kB active_file:28kB inactive_file:8kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165377.305275] lowmem_reserve[]: 0 0 13040 13040 [165377.305278] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:308 all_unreclaimable? yes [165377.305279] lowmem_reserve[]: 0 0 0 0 [165377.305282] Node 1 Normal free:47276kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:116kB active_file:92kB inactive_file:564kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:33756kB slab_unreclaimable:958788kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165377.305283] lowmem_reserve[]: 0 0 0 0 [165377.305288] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165377.305293] Node 0 DMA32: 93*4kB (UEM) 2344*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59604kB [165377.305297] Node 0 Normal: 244*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37152kB [165377.305302] Node 1 Normal: 566*4kB (UEM) 384*8kB (UEM) 212*16kB (UEM) 435*32kB (UM) 312*64kB (UM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46968kB [165377.305303] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165377.305303] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165377.305304] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165377.305304] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165377.305305] 1779 total pagecache pages [165377.305305] 52 pages in swap cache [165377.305306] Swap cache stats: add 311590, delete 311538, find 92805/149724 [165377.305307] Free swap = 16247860kB [165377.305307] Total swap = 16319484kB [165377.305308] 8369063 pages RAM [165377.305308] 0 pages HighMem/MovableOnly [165377.305308] 241261 pages reserved [165377.305504] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165377.305505] CPU: 0 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165377.305506] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165377.305509] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165377.305510] 00000000000080d0 00000000fa2ca73a ffff88073f6577c0 ffffffff816a3e2d [165377.305511] ffff88073f657850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165377.305512] ffffffffffffff00 000080d000000000 ffff88073f657820 00000000fa2ca73a [165377.305512] Call Trace: [165377.305514] [] dump_stack+0x19/0x1b [165377.305516] [] warn_alloc_failed+0x110/0x180 [165377.305517] [] ? drain_pages+0xb0/0xb0 [165377.305518] [] __alloc_pages_slowpath+0x6b6/0x724 [165377.305519] [] __alloc_pages_nodemask+0x405/0x420 [165377.305521] [] alloc_pages_current+0x98/0x110 [165377.305523] [] __get_free_pages+0xe/0x40 [165377.305524] [] swiotlb_alloc_coherent+0x5e/0x150 [165377.305526] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165377.305533] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165377.305538] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165377.305542] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165377.305546] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165377.305550] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165377.305555] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165377.305560] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165377.305562] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165377.305566] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165377.305569] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165377.305571] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165377.305573] [] process_one_work+0x17a/0x440 [165377.305574] [] worker_thread+0x126/0x3c0 [165377.305575] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165377.305576] [] kthread+0xcf/0xe0 [165377.305578] [] ? insert_kthread_work+0x40/0x40 [165377.305579] [] ret_from_fork+0x58/0x90 [165377.305580] [] ? insert_kthread_work+0x40/0x40 [165377.305580] Mem-Info: [165377.305584] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:89 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16196 slab_unreclaimable:564396 mapped:1543 shmem:0 pagetables:1561 bounce:0 free:39848 free_pcp:0 free_cma:0 [165377.305587] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165377.305588] lowmem_reserve[]: 0 2580 15620 15620 [165377.305592] Node 0 DMA32 free:59572kB min:7320kB low:9148kB high:10980kB active_anon:8kB inactive_anon:12kB active_file:28kB inactive_file:8kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:20kB mapped:344kB shmem:0kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165377.305593] lowmem_reserve[]: 0 0 13040 13040 [165377.305596] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101936kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:308 all_unreclaimable? yes [165377.305597] lowmem_reserve[]: 0 0 0 0 [165377.305601] Node 1 Normal free:47276kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:116kB active_file:92kB inactive_file:564kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:33756kB slab_unreclaimable:958788kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165377.305602] lowmem_reserve[]: 0 0 0 0 [165377.305607] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165377.305611] Node 0 DMA32: 93*4kB (UEM) 2344*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59604kB [165377.305615] Node 0 Normal: 244*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 280*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37152kB [165377.305619] Node 1 Normal: 566*4kB (UEM) 384*8kB (UEM) 212*16kB (UEM) 435*32kB (UM) 312*64kB (UM) 34*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46968kB [165377.305620] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165377.305621] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165377.305621] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165377.305622] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165377.305622] 1779 total pagecache pages [165377.305622] 52 pages in swap cache [165377.305623] Swap cache stats: add 311590, delete 311538, find 92805/149724 [165377.305623] Free swap = 16247860kB [165377.305624] Total swap = 16319484kB [165377.305624] 8369063 pages RAM [165377.305625] 0 pages HighMem/MovableOnly [165377.305625] 241261 pages reserved [165379.328484] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165379.328486] CPU: 21 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165379.328487] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165379.328491] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165379.328493] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165379.328494] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165379.328495] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165379.328495] Call Trace: [165379.328498] [] dump_stack+0x19/0x1b [165379.328500] [] warn_alloc_failed+0x110/0x180 [165379.328502] [] __alloc_pages_slowpath+0x6b6/0x724 [165379.328504] [] __alloc_pages_nodemask+0x405/0x420 [165379.328506] [] dma_generic_alloc_coherent+0x8f/0x140 [165379.328507] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165379.328516] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165379.328521] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165379.328527] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165379.328529] [] ? dma_pool_free+0xa7/0xd0 [165379.328534] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165379.328538] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165379.328542] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165379.328549] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165379.328551] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165379.328555] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165379.328559] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165379.328561] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165379.328563] [] process_one_work+0x17a/0x440 [165379.328564] [] worker_thread+0x126/0x3c0 [165379.328565] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165379.328567] [] kthread+0xcf/0xe0 [165379.328568] [] ? insert_kthread_work+0x40/0x40 [165379.328569] [] ret_from_fork+0x58/0x90 [165379.328570] [] ? insert_kthread_work+0x40/0x40 [165379.328571] Mem-Info: [165379.328575] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:68 inactive_file:115 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16220 slab_unreclaimable:564404 mapped:1563 shmem:1 pagetables:1561 bounce:0 free:39465 free_pcp:31 free_cma:0 [165379.328579] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165379.328580] lowmem_reserve[]: 0 2580 15620 15620 [165379.328584] Node 0 DMA32 free:59540kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:40kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:28kB mapped:344kB shmem:4kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no [165379.328585] lowmem_reserve[]: 0 0 13040 13040 [165379.328588] Node 0 Normal free:36928kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101968kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:308 all_unreclaimable? yes [165379.328589] lowmem_reserve[]: 0 0 0 0 [165379.328592] Node 1 Normal free:45812kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:140kB active_file:376kB inactive_file:676kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:24kB mapped:4400kB shmem:0kB slab_reclaimable:33852kB slab_unreclaimable:958788kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:297 all_unreclaimable? no [165379.328593] lowmem_reserve[]: 0 0 0 0 [165379.328598] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165379.328603] Node 0 DMA32: 89*4kB (UEM) 2346*8kB (UEM) 1290*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59572kB [165379.328607] Node 0 Normal: 244*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 279*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37120kB [165379.328611] Node 1 Normal: 491*4kB (UEM) 370*8kB (UEM) 215*16kB (UEM) 445*32kB (UEM) 312*64kB (UM) 33*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46796kB [165379.328612] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165379.328613] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165379.328613] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165379.328614] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165379.328614] 1882 total pagecache pages [165379.328615] 38 pages in swap cache [165379.328616] Swap cache stats: add 312294, delete 312256, find 93036/150125 [165379.328616] Free swap = 16247868kB [165379.328616] Total swap = 16319484kB [165379.328617] 8369063 pages RAM [165379.328617] 0 pages HighMem/MovableOnly [165379.328617] 241261 pages reserved [165379.339153] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165379.339156] CPU: 21 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165379.339157] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165379.339166] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165379.339168] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165379.339169] ffff880406203850 ffffffff81188820 0000000000000000 ffff88043ffdb000 [165379.339182] 0000000000000008 00000000000080d0 ffff880406203850 0000000039b5c568 [165379.339183] Call Trace: [165379.339188] [] dump_stack+0x19/0x1b [165379.339192] [] warn_alloc_failed+0x110/0x180 [165379.339193] [] __alloc_pages_slowpath+0x6b6/0x724 [165379.339195] [] __alloc_pages_nodemask+0x405/0x420 [165379.339197] [] alloc_pages_current+0x98/0x110 [165379.339199] [] __get_free_pages+0xe/0x40 [165379.339201] [] swiotlb_alloc_coherent+0x5e/0x150 [165379.339204] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165379.339224] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165379.339229] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165379.339242] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165379.339245] [] ? dma_pool_free+0xa7/0xd0 [165379.339249] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165379.339253] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165379.339258] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165379.339265] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165379.339267] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165379.339273] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165379.339278] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165379.339280] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165379.339283] [] process_one_work+0x17a/0x440 [165379.339284] [] worker_thread+0x126/0x3c0 [165379.339285] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165379.339286] [] kthread+0xcf/0xe0 [165379.339288] [] ? insert_kthread_work+0x40/0x40 [165379.339289] [] ret_from_fork+0x58/0x90 [165379.339290] [] ? insert_kthread_work+0x40/0x40 [165379.339291] Mem-Info: [165379.339298] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:67 inactive_file:116 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16220 slab_unreclaimable:564404 mapped:1563 shmem:1 pagetables:1561 bounce:0 free:39451 free_pcp:194 free_cma:0 [165379.339302] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165379.339304] lowmem_reserve[]: 0 2580 15620 15620 [165379.339307] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:36kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:28kB mapped:344kB shmem:4kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165379.339308] lowmem_reserve[]: 0 0 13040 13040 [165379.339312] Node 0 Normal free:36928kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101968kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:308 all_unreclaimable? yes [165379.339313] lowmem_reserve[]: 0 0 0 0 [165379.339317] Node 1 Normal free:45828kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:84kB active_file:376kB inactive_file:676kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4400kB shmem:0kB slab_reclaimable:33852kB slab_unreclaimable:958788kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:776kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:825 all_unreclaimable? no [165379.339318] lowmem_reserve[]: 0 0 0 0 [165379.339323] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165379.339327] Node 0 DMA32: 88*4kB (UEM) 2345*8kB (UEM) 1292*16kB (UEM) 460*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59560kB [165379.339331] Node 0 Normal: 244*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 279*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37120kB [165379.339335] Node 1 Normal: 301*4kB (UEM) 372*8kB (UEM) 215*16kB (UEM) 445*32kB (UEM) 312*64kB (UM) 33*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46052kB [165379.339337] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165379.339337] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165379.339338] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165379.339338] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165379.339339] 1868 total pagecache pages [165379.339339] 61 pages in swap cache [165379.339340] Swap cache stats: add 312317, delete 312256, find 93036/150125 [165379.339341] Free swap = 16247868kB [165379.339341] Total swap = 16319484kB [165379.339342] 8369063 pages RAM [165379.339342] 0 pages HighMem/MovableOnly [165379.339342] 241261 pages reserved [165381.365754] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165381.365756] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165381.365756] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165381.365761] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165381.365762] 00000000000080d0 0000000039b5c568 ffff880406203810 ffffffff816a3e2d [165381.365763] ffff8804062038a0 ffffffff81188820 0000000000000000 00000000ffffffff [165381.365764] ffffffffffffff00 000080d000000000 ffff880406203870 0000000039b5c568 [165381.365764] Call Trace: [165381.365767] [] dump_stack+0x19/0x1b [165381.365769] [] warn_alloc_failed+0x110/0x180 [165381.365770] [] __alloc_pages_slowpath+0x6b6/0x724 [165381.365771] [] __alloc_pages_nodemask+0x405/0x420 [165381.365775] [] dma_generic_alloc_coherent+0x8f/0x140 [165381.365776] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165381.365786] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165381.365791] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165381.365796] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165381.365798] [] ? dma_pool_free+0xa7/0xd0 [165381.365802] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165381.365806] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165381.365811] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165381.365817] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165381.365819] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165381.365823] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165381.365827] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165381.365829] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165381.365831] [] process_one_work+0x17a/0x440 [165381.365832] [] worker_thread+0x126/0x3c0 [165381.365834] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165381.365835] [] kthread+0xcf/0xe0 [165381.365836] [] ? insert_kthread_work+0x40/0x40 [165381.365837] [] ret_from_fork+0x58/0x90 [165381.365838] [] ? insert_kthread_work+0x40/0x40 [165381.365839] Mem-Info: [165381.365843] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:22 inactive_file:43 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16220 slab_unreclaimable:564408 mapped:1542 shmem:1 pagetables:1561 bounce:0 free:39775 free_pcp:0 free_cma:0 [165381.365846] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165381.365848] lowmem_reserve[]: 0 2580 15620 15620 [165381.365851] Node 0 DMA32 free:59588kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:16kB active_file:24kB inactive_file:12kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:28kB mapped:340kB shmem:4kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165381.365852] lowmem_reserve[]: 0 0 13040 13040 [165381.365856] Node 0 Normal free:36928kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101968kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:308 all_unreclaimable? yes [165381.365857] lowmem_reserve[]: 0 0 0 0 [165381.365860] Node 1 Normal free:47004kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:4kB active_file:208kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:33852kB slab_unreclaimable:958804kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165381.365861] lowmem_reserve[]: 0 0 0 0 [165381.365866] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165381.365871] Node 0 DMA32: 93*4kB (UEM) 2344*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59604kB [165381.365875] Node 0 Normal: 244*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 279*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37120kB [165381.365879] Node 1 Normal: 515*4kB (UEM) 374*8kB (UEM) 215*16kB (UEM) 444*32kB (UEM) 312*64kB (UM) 33*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46892kB [165381.365880] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165381.365881] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165381.365881] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165381.365882] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165381.365882] 1725 total pagecache pages [165381.365882] 35 pages in swap cache [165381.365883] Swap cache stats: add 312418, delete 312383, find 93068/150178 [165381.365884] Free swap = 16247864kB [165381.365884] Total swap = 16319484kB [165381.365885] 8369063 pages RAM [165381.365885] 0 pages HighMem/MovableOnly [165381.365885] 241261 pages reserved [165381.366114] kworker/u480:4: page allocation failure: order:8, mode:0x80d0 [165381.366115] CPU: 0 PID: 67406 Comm: kworker/u480:4 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165381.366116] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165381.366118] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165381.366119] 00000000000080d0 0000000039b5c568 ffff8804062037c0 ffffffff816a3e2d [165381.366120] ffff880406203850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165381.366121] ffffffffffffff00 000080d000000000 ffff880406203820 0000000039b5c568 [165381.366122] Call Trace: [165381.366123] [] dump_stack+0x19/0x1b [165381.366125] [] warn_alloc_failed+0x110/0x180 [165381.366126] [] ? drain_pages+0xb0/0xb0 [165381.366127] [] __alloc_pages_slowpath+0x6b6/0x724 [165381.366129] [] __alloc_pages_nodemask+0x405/0x420 [165381.366130] [] alloc_pages_current+0x98/0x110 [165381.366132] [] __get_free_pages+0xe/0x40 [165381.366133] [] swiotlb_alloc_coherent+0x5e/0x150 [165381.366135] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165381.366142] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165381.366147] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165381.366151] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165381.366153] [] ? dma_pool_free+0xa7/0xd0 [165381.366157] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165381.366161] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165381.366165] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165381.366170] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165381.366172] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165381.366176] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165381.366179] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165381.366181] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165381.366183] [] process_one_work+0x17a/0x440 [165381.366184] [] worker_thread+0x126/0x3c0 [165381.366185] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165381.366186] [] kthread+0xcf/0xe0 [165381.366188] [] ? insert_kthread_work+0x40/0x40 [165381.366189] [] ret_from_fork+0x58/0x90 [165381.366190] [] ? insert_kthread_work+0x40/0x40 [165381.366190] Mem-Info: [165381.366194] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:22 inactive_file:43 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16220 slab_unreclaimable:564408 mapped:1542 shmem:1 pagetables:1561 bounce:0 free:39775 free_pcp:0 free_cma:0 [165381.366197] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165381.366199] lowmem_reserve[]: 0 2580 15620 15620 [165381.366202] Node 0 DMA32 free:59588kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:16kB active_file:24kB inactive_file:12kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:28kB mapped:340kB shmem:4kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165381.366203] lowmem_reserve[]: 0 0 13040 13040 [165381.366207] Node 0 Normal free:36928kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101968kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:308 all_unreclaimable? yes [165381.366208] lowmem_reserve[]: 0 0 0 0 [165381.366211] Node 1 Normal free:47004kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:4kB active_file:208kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:33852kB slab_unreclaimable:958804kB kernel_stack:17392kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165381.366212] lowmem_reserve[]: 0 0 0 0 [165381.366217] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165381.366221] Node 0 DMA32: 93*4kB (UEM) 2344*8kB (UEM) 1292*16kB (UEM) 461*32kB (UM) 77*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59604kB [165381.366225] Node 0 Normal: 244*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 279*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37120kB [165381.366230] Node 1 Normal: 515*4kB (UEM) 374*8kB (UEM) 215*16kB (UEM) 444*32kB (UEM) 312*64kB (UM) 33*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46892kB [165381.366230] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165381.366231] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165381.366231] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165381.366232] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165381.366232] 1725 total pagecache pages [165381.366233] 35 pages in swap cache [165381.366233] Swap cache stats: add 312418, delete 312383, find 93068/150178 [165381.366234] Free swap = 16247864kB [165381.366234] Total swap = 16319484kB [165381.366235] 8369063 pages RAM [165381.366235] 0 pages HighMem/MovableOnly [165381.366235] 241261 pages reserved [165383.362803] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165383.362805] CPU: 6 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165383.362805] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165383.362815] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165383.362817] 00000000000080d0 00000000fa2ca73a ffff88073f657810 ffffffff816a3e2d [165383.362819] ffff88073f6578a0 ffffffff81188820 0000000000000000 00000000ffffffff [165383.362820] ffffffffffffff00 000080d000000000 ffff88073f657870 00000000fa2ca73a [165383.362821] Call Trace: [165383.362827] [] dump_stack+0x19/0x1b [165383.362832] [] warn_alloc_failed+0x110/0x180 [165383.362834] [] __alloc_pages_slowpath+0x6b6/0x724 [165383.362836] [] __alloc_pages_nodemask+0x405/0x420 [165383.362839] [] dma_generic_alloc_coherent+0x8f/0x140 [165383.362841] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165383.362864] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165383.362870] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165383.362877] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165383.362879] [] ? dma_pool_free+0xa7/0xd0 [165383.362884] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165383.362890] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165383.362896] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165383.362906] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165383.362909] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165383.362915] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165383.362921] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165383.362924] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165383.362927] [] process_one_work+0x17a/0x440 [165383.362929] [] worker_thread+0x126/0x3c0 [165383.362931] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165383.362933] [] kthread+0xcf/0xe0 [165383.362935] [] ? insert_kthread_work+0x40/0x40 [165383.362938] [] ret_from_fork+0x58/0x90 [165383.362939] [] ? insert_kthread_work+0x40/0x40 [165383.362940] Mem-Info: [165383.362947] active_anon:9 inactive_anon:0 isolated_anon:30 active_file:30 inactive_file:44 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16253 slab_unreclaimable:564408 mapped:1542 shmem:1 pagetables:1561 bounce:0 free:39561 free_pcp:27 free_cma:0 [165383.362952] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165383.362953] lowmem_reserve[]: 0 2580 15620 15620 [165383.362957] Node 0 DMA32 free:59496kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:28kB mapped:340kB shmem:4kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:96kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:38 all_unreclaimable? no [165383.362958] lowmem_reserve[]: 0 0 13040 13040 [165383.362962] Node 0 Normal free:36928kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101968kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:308 all_unreclaimable? yes [165383.362964] lowmem_reserve[]: 0 0 0 0 [165383.362970] Node 1 Normal free:46240kB min:45728kB low:57160kB high:68592kB active_anon:36kB inactive_anon:148kB active_file:248kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:20kB mapped:4320kB shmem:0kB slab_reclaimable:33984kB slab_unreclaimable:958804kB kernel_stack:17408kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:108kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:149 all_unreclaimable? no [165383.362971] lowmem_reserve[]: 0 0 0 0 [165383.362979] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165383.362986] Node 0 DMA32: 91*4kB (UE) 2342*8kB (UE) 1293*16kB (UEM) 456*32kB (UM) 78*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59500kB [165383.362992] Node 0 Normal: 244*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 279*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37120kB [165383.362996] Node 1 Normal: 523*4kB (UEM) 346*8kB (EM) 211*16kB (UEM) 443*32kB (UEM) 312*64kB (UM) 32*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46476kB [165383.362997] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165383.362998] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165383.362998] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165383.362999] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165383.362999] 1785 total pagecache pages [165383.363000] 63 pages in swap cache [165383.363001] Swap cache stats: add 313333, delete 313270, find 93342/150663 [165383.363001] Free swap = 16247868kB [165383.363002] Total swap = 16319484kB [165383.363002] 8369063 pages RAM [165383.363002] 0 pages HighMem/MovableOnly [165383.363003] 241261 pages reserved [165383.363311] kworker/u480:1: page allocation failure: order:8, mode:0x80d0 [165383.363312] CPU: 6 PID: 67604 Comm: kworker/u480:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165383.363313] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165383.363316] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165383.363317] 00000000000080d0 00000000fa2ca73a ffff88073f6577c0 ffffffff816a3e2d [165383.363318] ffff88073f657850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165383.363319] ffffffffffffff00 000080d000000000 ffff88073f657820 00000000fa2ca73a [165383.363319] Call Trace: [165383.363321] [] dump_stack+0x19/0x1b [165383.363324] [] warn_alloc_failed+0x110/0x180 [165383.363325] [] ? drain_pages+0xb0/0xb0 [165383.363327] [] __alloc_pages_slowpath+0x6b6/0x724 [165383.363329] [] __alloc_pages_nodemask+0x405/0x420 [165383.363332] [] alloc_pages_current+0x98/0x110 [165383.363334] [] __get_free_pages+0xe/0x40 [165383.363338] [] swiotlb_alloc_coherent+0x5e/0x150 [165383.363340] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165383.363348] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165383.363354] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165383.363360] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165383.363362] [] ? dma_pool_free+0xa7/0xd0 [165383.363368] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165383.363373] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165383.363379] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165383.363384] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165383.363386] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165383.363390] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165383.363394] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165383.363397] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165383.363399] [] process_one_work+0x17a/0x440 [165383.363401] [] worker_thread+0x126/0x3c0 [165383.363403] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165383.363405] [] kthread+0xcf/0xe0 [165383.363406] [] ? insert_kthread_work+0x40/0x40 [165383.363408] [] ret_from_fork+0x58/0x90 [165383.363410] [] ? insert_kthread_work+0x40/0x40 [165383.363411] Mem-Info: [165383.363417] active_anon:9 inactive_anon:0 isolated_anon:0 active_file:30 inactive_file:44 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:16253 slab_unreclaimable:564408 mapped:1542 shmem:1 pagetables:1561 bounce:0 free:39576 free_pcp:0 free_cma:0 [165383.363421] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165383.363423] lowmem_reserve[]: 0 2580 15620 15620 [165383.363426] Node 0 DMA32 free:59556kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:28kB mapped:340kB shmem:4kB slab_reclaimable:4284kB slab_unreclaimable:196860kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no [165383.363427] lowmem_reserve[]: 0 0 13040 13040 [165383.363432] Node 0 Normal free:36928kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1508kB shmem:0kB slab_reclaimable:26744kB slab_unreclaimable:1101968kB kernel_stack:15568kB pagetables:2268kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:308 all_unreclaimable? yes [165383.363433] lowmem_reserve[]: 0 0 0 0 [165383.363439] Node 1 Normal free:46240kB min:45728kB low:57160kB high:68592kB active_anon:36kB inactive_anon:148kB active_file:248kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:20kB mapped:4320kB shmem:0kB slab_reclaimable:33984kB slab_unreclaimable:958804kB kernel_stack:17408kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:138 all_unreclaimable? no [165383.363440] lowmem_reserve[]: 0 0 0 0 [165383.363448] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165383.363455] Node 0 DMA32: 95*4kB (UEM) 2346*8kB (UEM) 1294*16kB (UEM) 457*32kB (UM) 78*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59596kB [165383.363460] Node 0 Normal: 244*4kB (UEM) 174*8kB (UEM) 210*16kB (UEM) 279*32kB (UM) 315*64kB (UEM) 18*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37120kB [165383.363467] Node 1 Normal: 546*4kB (UEM) 350*8kB (EM) 211*16kB (UEM) 443*32kB (UEM) 312*64kB (UM) 32*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46600kB [165383.363468] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165383.363469] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165383.363470] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165383.363471] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165383.363472] 1785 total pagecache pages [165383.363472] 59 pages in swap cache [165383.363473] Swap cache stats: add 313335, delete 313276, find 93342/150663 [165383.363474] Free swap = 16247868kB [165383.363474] Total swap = 16319484kB [165383.363475] 8369063 pages RAM [165383.363476] 0 pages HighMem/MovableOnly [165383.363476] 241261 pages reserved [165398.026544] INFO: rcu_sched detected stalls on CPUs/tasks: { 18} (detected by 20, t=240007 jiffies, g=4033261, c=4033260, q=99409) [165398.026545] Task dump for CPU 18: [165398.026547] systemd-logind R running task 0 67495 1 0x0000008e [165398.026548] ffffffff81d86128 ffffffffffffffff ffff88073359ba58 ffffffff819041d2 [165398.026549] ffffffff81d8612e ffff88073359ba48 ffffffff8132f714 ffff88073359ffff [165398.026550] ffffffff81395475 0000000000000002 ffffffff81d86120 000000007e279edf [165398.026551] Call Trace: [165398.026556] [] ? vsnprintf+0x234/0x6a0 [165398.026558] [] ? fbcon_scroll+0x215/0xcf0 [165398.026560] [] ? __const_udelay+0x27/0x30 [165398.026562] [] ? wait_for_xmitr+0x40/0xa0 [165398.026564] [] ? serial8250_console_putchar+0x1c/0x30 [165398.026565] [] ? serial8250_console_write+0x140/0x140 [165398.026567] [] ? uart_console_write+0x3a/0x80 [165398.026568] [] ? serial8250_console_write+0xae/0x140 [165398.026571] [] ? call_console_drivers.constprop.17+0x90/0xf0 [165398.026573] [] ? console_unlock+0x468/0x4b0 [165398.026575] [] ? con_shutdown+0x28/0x30 [165398.026577] [] ? release_tty+0x37/0x140 [165398.026578] [] ? tty_release+0x370/0x530 [165398.026581] [] ? dentry_kill+0x154/0x1b0 [165398.026583] [] ? __fput+0xe9/0x260 [165398.026584] [] ? ____fput+0xe/0x10 [165398.026586] [] ? task_work_run+0xc5/0xf0 [165398.026588] [] ? do_exit+0x2d1/0xa40 [165398.026589] [] ? terminate_walk+0x49/0x50 [165398.026590] [] ? do_last+0x66d/0x12c0 [165398.026591] [] ? do_group_exit+0x3f/0xa0 [165398.026593] [] ? get_signal_to_deliver+0x1ce/0x5e0 [165398.026595] [] ? do_signal+0x57/0x6c0 [165398.026596] [] ? do_filp_open+0x4b/0xb0 [165398.026599] [] ? kmem_cache_free+0x1e2/0x200 [165398.026600] [] ? do_notify_resume+0x5f/0xb0 [165398.026602] [] ? int_signal+0x12/0x17 [165424.895604] Pid: 34752, comm: ll_ost01_024 [165424.895605] Call Trace: [165424.895611] [] schedule_preempt_disabled+0x29/0x70 [165424.895614] [] __mutex_lock_slowpath+0xc7/0x1d0 [165424.895616] [] mutex_lock+0x1f/0x2f [165424.895626] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [165424.895678] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [165424.895705] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [165424.895731] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [165424.895767] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [165424.895795] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [165424.895798] [] ? default_wake_function+0x12/0x20 [165424.895801] [] ? __wake_up_common+0x58/0x90 [165424.895828] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [165424.895854] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [165424.895856] [] kthread+0xcf/0xe0 [165424.895857] [] ? kthread+0x0/0xe0 [165424.895859] [] ret_from_fork+0x58/0x90 [165424.895860] [] ? kthread+0x0/0xe0 [165424.895860] [165424.895862] LustreError: dumping log to /tmp/lustre-log.1512324751.34752 [165454.660588] LNet: 5080:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.110@o2ib: 6 seconds [165454.660590] LNet: 5080:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 27 previous similar messages [165465.342567] Lustre: soaked-OST0016: Export ffff8806c1bc8400 already connecting from 192.168.1.111@o2ib [165465.342570] Lustre: Skipped 23 previous similar messages [165551.771266] Lustre: soaked-OST0016: Connection restored to (at 192.168.1.123@o2ib) [165551.771267] Lustre: Skipped 34 previous similar messages [165578.025289] INFO: rcu_sched detected stalls on CPUs/tasks: { 18} (detected by 20, t=420012 jiffies, g=4033261, c=4033260, q=101665) [165578.025290] Task dump for CPU 18: [165578.025292] systemd-logind R running task 0 67495 1 0x0000008e [165578.025293] ffffffff81d86128 ffffffffffffffff ffff88073359ba58 ffffffff819041d2 [165578.025294] ffffffff81d8612e ffff88073359ba48 ffffffff8132f714 ffff88073359ffff [165578.025295] ffffffff81395475 0000000000000002 ffffffff81d86120 000000007e279edf [165578.025295] Call Trace: [165578.025299] [] ? vsnprintf+0x234/0x6a0 [165578.025301] [] ? fbcon_scroll+0x215/0xcf0 [165578.025303] [] ? __const_udelay+0x27/0x30 [165578.025305] [] ? wait_for_xmitr+0x40/0xa0 [165578.025307] [] ? serial8250_console_putchar+0x1c/0x30 [165578.025309] [] ? serial8250_console_write+0x140/0x140 [165578.025310] [] ? uart_console_write+0x3a/0x80 [165578.025312] [] ? serial8250_console_write+0xae/0x140 [165578.025315] [] ? call_console_drivers.constprop.17+0x90/0xf0 [165578.025317] [] ? console_unlock+0x468/0x4b0 [165578.025320] [] ? con_shutdown+0x28/0x30 [165578.025321] [] ? release_tty+0x37/0x140 [165578.025323] [] ? tty_release+0x370/0x530 [165578.025325] [] ? dentry_kill+0x154/0x1b0 [165578.025327] [] ? __fput+0xe9/0x260 [165578.025329] [] ? ____fput+0xe/0x10 [165578.025332] [] ? task_work_run+0xc5/0xf0 [165578.025333] [] ? do_exit+0x2d1/0xa40 [165578.025335] [] ? terminate_walk+0x49/0x50 [165578.025336] [] ? do_last+0x66d/0x12c0 [165578.025338] [] ? do_group_exit+0x3f/0xa0 [165578.025340] [] ? get_signal_to_deliver+0x1ce/0x5e0 [165578.025342] [] ? do_signal+0x57/0x6c0 [165578.025343] [] ? do_filp_open+0x4b/0xb0 [165578.025346] [] ? kmem_cache_free+0x1e2/0x200 [165578.025347] [] ? do_notify_resume+0x5f/0xb0 [165578.025350] [] ? int_signal+0x12/0x17 [165629.823517] Lustre: 5090:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1512324901/real 0] req@ffff880806128000 x1585614571784768/t0(0) o38->soaked-MDT0002-lwp-OST0010@192.168.1.110@o2ib:12/10 lens 520/544 e 0 to 1 dl 1512324956 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [165629.823519] Lustre: 5090:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 22 previous similar messages [165683.958619] Lustre: 226541:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807f62ad050 x1585626139557456/t0(0) o4->1c674fc7-c4ea-661f-8bc3-4139e5a9579f@192.168.1.143@o2ib:370/0 lens 608/0 e 0 to 0 dl 1512325015 ref 2 fl New:/2/ffffffff rc 0/-1 [165683.958621] Lustre: 226541:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 335 previous similar messages [165704.665203] Lustre: 226541:0:(service.c:1961:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 11s req@ffff8804098e8050 x1585626138129328/t0(0) o4->66279ba2-1125-9263-dc81-36c68d2c363f@192.168.1.138@o2ib:0/0 lens 608/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 [165704.748395] Lustre: 226541:0:(service.c:1961:ptlrpc_server_handle_req_in()) Skipped 21 previous similar messages [165704.751264] Lustre: ost_io: This server is not able to keep up with request traffic (cpu-bound). [165704.751266] Lustre: 224968:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=27 reqQ=8 recA=119, svcEst=600, delay=10709(jiff) [165704.751272] Lustre: 224968:0:(service.c:1296:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8804120a2450 x1585626318055200/t0(0) o4->720bb4bb-27e0-0d4b-e87f-df72816138f7@192.168.1.125@o2ib:385/0 lens 608/0 e 0 to 0 dl 1512325030 ref 2 fl New:/2/ffffffff rc 0/-1 [165704.893998] ll_ost_out01_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [165704.894000] ll_ost_out01_00 cpuset=/ mems_allowed=1 [165704.894003] CPU: 24 PID: 183717 Comm: ll_ost_out01_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165704.894004] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165704.894007] ffff8803dd56eeb0 0000000085bdad45 ffff8803966af998 ffffffff816a3e2d [165704.894008] ffff8803966afa28 ffffffff8169f246 ffffffff810e93ac ffff8807f1e228a0 [165704.894009] ffff8807f1e228b8 0000000000000202 0000000000000202 ffff8803966afa18 [165704.894010] Call Trace: [165704.894020] [] dump_stack+0x19/0x1b [165704.894022] [] dump_header+0x90/0x229 [165704.894026] [] ? ktime_get_ts64+0x4c/0xf0 [165704.894030] [] ? delayacct_end+0x8f/0xb0 [165704.894035] [] oom_kill_process+0x254/0x3d0 [165704.894037] [] ? oom_unkillable_task+0xcd/0x120 [165704.894038] [] ? find_lock_task_mm+0x56/0xc0 [165704.894040] [] out_of_memory+0x4b6/0x4f0 [165704.894042] [] __alloc_pages_slowpath+0x5d6/0x724 [165704.894044] [] __alloc_pages_nodemask+0x405/0x420 [165704.894047] [] __vmalloc_node_range+0x12f/0x270 [165704.894103] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165704.894104] [] vzalloc_node+0x47/0x50 [165704.894133] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165704.894161] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165704.894188] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [165704.894217] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [165704.894245] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [165704.894248] [] kthread+0xcf/0xe0 [165704.894249] [] ? insert_kthread_work+0x40/0x40 [165704.894251] [] ret_from_fork+0x58/0x90 [165704.894252] [] ? insert_kthread_work+0x40/0x40 [165704.894253] Mem-Info: [165704.894258] active_anon:0 inactive_anon:30 isolated_anon:0 active_file:36 inactive_file:184 isolated_file:0 unevictable:6774 dirty:0 writeback:25 unstable:0 slab_reclaimable:13481 slab_unreclaimable:560768 mapped:1547 shmem:1 pagetables:1546 bounce:0 free:39374 free_pcp:0 free_cma:0 [165704.894264] Node 1 Normal free:45500kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:100kB active_file:240kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:96kB mapped:4316kB shmem:0kB slab_reclaimable:28296kB slab_unreclaimable:957604kB kernel_stack:17408kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1141 all_unreclaimable? yes [165704.894266] lowmem_reserve[]: 0 0 0 0 [165704.894272] Node 1 Normal: 546*4kB (UEM) 378*8kB (EM) 209*16kB (EM) 422*32kB (UEM) 346*64kB (UEM) 26*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47528kB [165704.894274] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165704.894275] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165704.894276] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165704.894276] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165704.894277] 1810 total pagecache pages [165704.894278] 100 pages in swap cache [165704.894279] Swap cache stats: add 313452, delete 313352, find 93363/150709 [165704.894279] Free swap = 16247860kB [165704.894280] Total swap = 16319484kB [165704.894281] 8369063 pages RAM [165704.894281] 0 pages HighMem/MovableOnly [165704.894281] 241261 pages reserved [165704.894282] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165704.894290] [ 977] 0 977 20596 239 41 95 0 systemd-journal [165704.894292] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165704.894293] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165704.894295] [ 1046] 0 1046 1588718 3689 132 0 -1000 multipathd [165704.894303] [ 2373] 0 2373 13863 112 27 134 -1000 auditd [165704.894304] [ 2428] 0 2428 63244 275 58 617 0 rsyslogd [165704.894306] [ 2429] 0 2429 64920 329 78 271 0 sssd [165704.894307] [ 2434] 0 2434 67755 196 45 190 0 gssproxy [165704.894308] [ 2435] 0 2435 5461 183 15 171 0 irqbalance [165704.894310] [ 2444] 81 2444 9196 196 21 156 -900 dbus-daemon [165704.894311] [ 2481] 0 2481 136536 286 83 810 0 NetworkManager [165704.894313] [ 2488] 999 2488 136594 259 59 1650 0 polkitd [165704.894314] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165704.894316] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165704.894317] [ 2771] 998 2771 29553 227 29 138 0 chronyd [165704.894319] [ 2798] 0 2798 93411 322 125 640 0 sssd_be [165704.894320] [ 2828] 0 2828 66338 277 85 255 0 sssd_nss [165704.894322] [ 2829] 0 2829 61252 272 75 247 0 sssd_pam [165704.894323] [ 2830] 0 2830 59613 268 71 228 0 sssd_ssh [165704.894324] [ 2831] 0 2831 68163 264 86 329 0 sssd_pac [165704.894327] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165704.894328] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165704.894330] [ 3703] 996 3703 24211 162 21 147 0 munged [165704.894331] [ 3875] 0 3875 22907 180 44 269 0 master [165704.894332] [ 3908] 89 3908 25491 220 47 271 0 qmgr [165704.894333] [ 3915] 0 3915 31559 198 18 163 0 crond [165704.894335] [ 3919] 0 3919 168252 211 70 385 0 automount [165704.894336] [ 3951] 0 3951 6464 159 19 53 0 atd [165704.894337] [ 3970] 0 3970 27511 156 12 32 0 agetty [165704.894339] [ 3971] 0 3971 27511 160 10 31 0 agetty [165704.894363] [67360] 89 67360 25474 220 47 258 0 pickup [165704.894365] [67627] 0 67627 40829 126 33 208 0 crond [165704.894367] Out of memory: Kill process 2488 (polkitd) score 0 or sacrifice child [165704.894368] Killed process 2488 (polkitd) total-vm:546376kB, anon-rss:0kB, file-rss:1036kB, shmem-rss:0kB [165705.183471] qmgr invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [165705.183473] qmgr cpuset=/ mems_allowed=0-1 [165705.183475] CPU: 27 PID: 3908 Comm: qmgr Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165705.183476] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165705.183478] ffff88081ddb8000 0000000054c8457a ffff8808217bb920 ffffffff816a3e2d [165705.183480] ffff8808217bb9b0 ffffffff8169f246 ffffffff810e93ac ffff8807fde3a910 [165705.183482] ffff8807fde3a928 0000000000000202 0000000000000202 ffff8808217bb9a0 [165705.183483] Call Trace: [165705.183487] [] dump_stack+0x19/0x1b [165705.183489] [] dump_header+0x90/0x229 [165705.183491] [] ? ktime_get_ts64+0x4c/0xf0 [165705.183493] [] ? delayacct_end+0x8f/0xb0 [165705.183496] [] oom_kill_process+0x254/0x3d0 [165705.183498] [] ? oom_unkillable_task+0xcd/0x120 [165705.183500] [] ? find_lock_task_mm+0x56/0xc0 [165705.183502] [] out_of_memory+0x4b6/0x4f0 [165705.183504] [] __alloc_pages_slowpath+0x5d6/0x724 [165705.183507] [] __alloc_pages_nodemask+0x405/0x420 [165705.183510] [] alloc_pages_current+0x98/0x110 [165705.183512] [] __page_cache_alloc+0x97/0xb0 [165705.183514] [] filemap_fault+0x170/0x410 [165705.183544] [] ext4_filemap_fault+0x36/0x50 [ext4] [165705.183547] [] __do_fault+0x52/0xe0 [165705.183548] [] do_read_fault.isra.44+0x4b/0x130 [165705.183550] [] handle_mm_fault+0x691/0xfa0 [165705.183553] [] ? change_protection+0x65/0xa0 [165705.183557] [] __do_page_fault+0x154/0x450 [165705.183559] [] do_page_fault+0x35/0x90 [165705.183562] [] page_fault+0x28/0x30 [165705.183563] Mem-Info: [165705.183569] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:182 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:25 unstable:0 slab_reclaimable:13481 slab_unreclaimable:560768 mapped:1547 shmem:1 pagetables:1546 bounce:0 free:39374 free_pcp:127 free_cma:0 [165705.183576] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165705.183578] lowmem_reserve[]: 0 2580 15620 15620 [165705.183584] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:20kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:4kB mapped:340kB shmem:4kB slab_reclaimable:3308kB slab_unreclaimable:195132kB kernel_stack:1696kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:52kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4217 all_unreclaimable? yes [165705.183586] lowmem_reserve[]: 0 0 13040 13040 [165705.183592] Node 0 Normal free:36936kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:472kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:22320kB slab_unreclaimable:1090336kB kernel_stack:15568kB pagetables:2208kB unstable:0kB bounce:0kB free_pcp:140kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2330 all_unreclaimable? yes [165705.183593] lowmem_reserve[]: 0 0 0 0 [165705.183600] Node 1 Normal free:45500kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:100kB active_file:240kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:96kB mapped:4316kB shmem:0kB slab_reclaimable:28296kB slab_unreclaimable:957604kB kernel_stack:17408kB pagetables:3736kB unstable:0kB bounce:0kB free_pcp:316kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4037 all_unreclaimable? yes [165705.183601] lowmem_reserve[]: 0 0 0 0 [165705.183610] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165705.183618] Node 0 DMA32: 36*4kB (UEM) 2015*8kB (UE) 1296*16kB (UEM) 512*32kB (UEM) 97*64kB (UEM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59720kB [165705.183625] Node 0 Normal: 123*4kB (UEM) 145*8kB (UEM) 119*16kB (UEM) 311*32kB (UEM) 383*64kB (UEM) 12*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 39556kB [165705.183632] Node 1 Normal: 543*4kB (EM) 387*8kB (UEM) 209*16kB (EM) 419*32kB (UEM) 347*64kB (UEM) 26*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 47556kB [165705.183634] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165705.183635] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165705.183636] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165705.183637] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165705.183637] 1810 total pagecache pages [165705.183638] 25 pages in swap cache [165705.183639] Swap cache stats: add 313521, delete 313496, find 93404/150770 [165705.183640] Free swap = 16254380kB [165705.183641] Total swap = 16319484kB [165705.183641] 8369063 pages RAM [165705.183642] 0 pages HighMem/MovableOnly [165705.183642] 241261 pages reserved [165705.183643] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165705.183652] [ 977] 0 977 20596 238 41 95 0 systemd-journal [165705.183654] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165705.183656] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165705.183657] [ 1046] 0 1046 1588718 3689 132 0 -1000 multipathd [165705.183667] [ 2373] 0 2373 13863 112 27 134 -1000 auditd [165705.183668] [ 2428] 0 2428 63244 275 58 620 0 rsyslogd [165705.183670] [ 2429] 0 2429 64920 329 78 273 0 sssd [165705.183671] [ 2434] 0 2434 67755 211 45 169 0 gssproxy [165705.183673] [ 2435] 0 2435 5461 183 15 172 0 irqbalance [165705.183675] [ 2444] 81 2444 9196 196 21 156 -900 dbus-daemon [165705.183676] [ 2481] 0 2481 136536 286 83 813 0 NetworkManager [165705.183678] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165705.183680] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165705.183682] [ 2771] 998 2771 29553 227 29 141 0 chronyd [165705.183684] [ 2798] 0 2798 93411 322 125 641 0 sssd_be [165705.183685] [ 2828] 0 2828 66338 277 85 257 0 sssd_nss [165705.183687] [ 2829] 0 2829 61252 271 75 248 0 sssd_pam [165705.183688] [ 2830] 0 2830 59613 268 71 230 0 sssd_ssh [165705.183690] [ 2831] 0 2831 68163 263 86 329 0 sssd_pac [165705.183692] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165705.183694] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165705.183696] [ 3703] 996 3703 24211 162 21 149 0 munged [165705.183697] [ 3875] 0 3875 22907 180 44 271 0 master [165705.183699] [ 3908] 89 3908 25491 220 47 272 0 qmgr [165705.183701] [ 3915] 0 3915 31559 198 18 164 0 crond [165705.183702] [ 3919] 0 3919 168252 211 70 387 0 automount [165705.183704] [ 3951] 0 3951 6464 159 19 54 0 atd [165705.183706] [ 3970] 0 3970 27511 156 12 32 0 agetty [165705.183707] [ 3971] 0 3971 27511 160 10 31 0 agetty [165705.183735] [67360] 89 67360 25474 220 47 259 0 pickup [165705.183737] [67627] 0 67627 40829 126 33 208 0 crond [165705.183739] Out of memory: Kill process 2481 (NetworkManager) score 0 or sacrifice child [165705.183741] Killed process 2481 (NetworkManager) total-vm:546144kB, anon-rss:0kB, file-rss:1144kB, shmem-rss:0kB [165706.285420] kworker/u480:2: page allocation failure: order:8, mode:0x80d0 [165706.285422] CPU: 0 PID: 67636 Comm: kworker/u480:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165706.285423] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165706.285432] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165706.285433] 00000000000080d0 00000000987a9902 ffff88052cdf7810 ffffffff816a3e2d [165706.285434] ffff88052cdf78a0 ffffffff81188820 0000000000000000 ffff88043ffdb000 [165706.285435] 0000000000000008 00000000000080d0 ffff88052cdf78a0 00000000987a9902 [165706.285436] Call Trace: [165706.285441] [] dump_stack+0x19/0x1b [165706.285444] [] warn_alloc_failed+0x110/0x180 [165706.285446] [] __alloc_pages_slowpath+0x6b6/0x724 [165706.285447] [] __alloc_pages_nodemask+0x405/0x420 [165706.285450] [] dma_generic_alloc_coherent+0x8f/0x140 [165706.285452] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165706.285470] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165706.285475] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165706.285481] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165706.285486] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165706.285490] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165706.285494] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165706.285502] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165706.285504] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165706.285511] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165706.285516] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165706.285519] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165706.285521] [] process_one_work+0x17a/0x440 [165706.285522] [] worker_thread+0x126/0x3c0 [165706.285523] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165706.285525] [] kthread+0xcf/0xe0 [165706.285526] [] ? insert_kthread_work+0x40/0x40 [165706.285528] [] ret_from_fork+0x58/0x90 [165706.285529] [] ? insert_kthread_work+0x40/0x40 [165706.285529] Mem-Info: [165706.285534] active_anon:0 inactive_anon:26 isolated_anon:0 active_file:147 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:2 unstable:0 slab_reclaimable:12315 slab_unreclaimable:560524 mapped:1547 shmem:0 pagetables:1406 bounce:0 free:39222 free_pcp:110 free_cma:0 [165706.285539] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165706.285540] lowmem_reserve[]: 0 2580 15620 15620 [165706.285544] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:32kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3164kB slab_unreclaimable:195116kB kernel_stack:1568kB pagetables:212kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no [165706.285545] lowmem_reserve[]: 0 0 13040 13040 [165706.285549] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:112kB active_file:104kB inactive_file:112kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1528kB shmem:0kB slab_reclaimable:19600kB slab_unreclaimable:1089508kB kernel_stack:14192kB pagetables:2148kB unstable:0kB bounce:0kB free_pcp:368kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:277 all_unreclaimable? no [165706.285550] lowmem_reserve[]: 0 0 0 0 [165706.285553] Node 1 Normal free:45228kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:28kB active_file:484kB inactive_file:0kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:8kB mapped:4320kB shmem:0kB slab_reclaimable:26496kB slab_unreclaimable:957472kB kernel_stack:16768kB pagetables:3264kB unstable:0kB bounce:0kB free_pcp:72kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:264 all_unreclaimable? no [165706.285554] lowmem_reserve[]: 0 0 0 0 [165706.285560] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165706.285564] Node 0 DMA32: 37*4kB (UEM) 2001*8kB (UEM) 1297*16kB (UEM) 512*32kB (UEM) 96*64kB (UEM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165706.285569] Node 0 Normal: 60*4kB (EM) 145*8kB (EM) 114*16kB (UEM) 273*32kB (UEM) 383*64kB (UEM) 12*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 38008kB [165706.285575] Node 1 Normal: 513*4kB (UEM) 386*8kB (EM) 213*16kB (UEM) 385*32kB (UEM) 347*64kB (UEM) 26*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46404kB [165706.285577] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165706.285578] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165706.285578] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165706.285580] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165706.285580] 1827 total pagecache pages [165706.285581] 44 pages in swap cache [165706.285582] Swap cache stats: add 313655, delete 313611, find 93432/150823 [165706.285583] Free swap = 16257452kB [165706.285583] Total swap = 16319484kB [165706.285584] 8369063 pages RAM [165706.285585] 0 pages HighMem/MovableOnly [165706.285585] 241261 pages reserved [165706.285900] kworker/u480:2: page allocation failure: order:8, mode:0x80d0 [165706.285901] CPU: 0 PID: 67636 Comm: kworker/u480:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165706.285902] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165706.285906] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165706.285907] 00000000000080d0 00000000987a9902 ffff88052cdf77c0 ffffffff816a3e2d [165706.285909] ffff88052cdf7850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165706.285911] ffffffffffffff00 000080d000000000 ffff88052cdf7820 00000000987a9902 [165706.285911] Call Trace: [165706.285913] [] dump_stack+0x19/0x1b [165706.285915] [] warn_alloc_failed+0x110/0x180 [165706.285917] [] ? drain_pages+0xb0/0xb0 [165706.285918] [] __alloc_pages_slowpath+0x6b6/0x724 [165706.285920] [] __alloc_pages_nodemask+0x405/0x420 [165706.285922] [] alloc_pages_current+0x98/0x110 [165706.285924] [] __get_free_pages+0xe/0x40 [165706.285928] [] swiotlb_alloc_coherent+0x5e/0x150 [165706.285929] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165706.285938] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165706.285944] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165706.285950] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165706.285955] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165706.285959] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165706.285965] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165706.285971] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165706.285974] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165706.285978] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165706.285983] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165706.285985] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165706.285987] [] process_one_work+0x17a/0x440 [165706.285989] [] worker_thread+0x126/0x3c0 [165706.285991] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165706.285992] [] kthread+0xcf/0xe0 [165706.285994] [] ? insert_kthread_work+0x40/0x40 [165706.285995] [] ret_from_fork+0x58/0x90 [165706.285997] [] ? insert_kthread_work+0x40/0x40 [165706.285997] Mem-Info: [165706.286002] active_anon:0 inactive_anon:26 isolated_anon:0 active_file:126 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:2 unstable:0 slab_reclaimable:12315 slab_unreclaimable:560524 mapped:1548 shmem:0 pagetables:1406 bounce:0 free:39253 free_pcp:0 free_cma:0 [165706.286007] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165706.286009] lowmem_reserve[]: 0 2580 15620 15620 [165706.286014] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:8kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3164kB slab_unreclaimable:195116kB kernel_stack:1568kB pagetables:212kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:54 all_unreclaimable? no [165706.286015] lowmem_reserve[]: 0 0 13040 13040 [165706.286020] Node 0 Normal free:36772kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:112kB active_file:120kB inactive_file:96kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19600kB slab_unreclaimable:1089508kB kernel_stack:14192kB pagetables:2148kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:108 all_unreclaimable? no [165706.286021] lowmem_reserve[]: 0 0 0 0 [165706.286026] Node 1 Normal free:45228kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:28kB active_file:360kB inactive_file:0kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:8kB mapped:4320kB shmem:0kB slab_reclaimable:26496kB slab_unreclaimable:957472kB kernel_stack:16768kB pagetables:3264kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:119 all_unreclaimable? no [165706.286027] lowmem_reserve[]: 0 0 0 0 [165706.286034] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165706.286040] Node 0 DMA32: 37*4kB (UEM) 2001*8kB (UEM) 1297*16kB (UEM) 512*32kB (UEM) 96*64kB (UEM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165706.286046] Node 0 Normal: 120*4kB (UEM) 145*8kB (UEM) 112*16kB (EM) 278*32kB (UEM) 383*64kB (UEM) 12*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 38376kB [165706.286053] Node 1 Normal: 534*4kB (UEM) 386*8kB (EM) 213*16kB (UEM) 385*32kB (UEM) 347*64kB (UEM) 26*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46488kB [165706.286054] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165706.286055] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165706.286056] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165706.286056] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165706.286057] 1827 total pagecache pages [165706.286057] 42 pages in swap cache [165706.286058] Swap cache stats: add 313656, delete 313614, find 93432/150823 [165706.286059] Free swap = 16257448kB [165706.286059] Total swap = 16319484kB [165706.286060] 8369063 pages RAM [165706.286060] 0 pages HighMem/MovableOnly [165706.286061] 241261 pages reserved [165713.943994] Pid: 222336, comm: ll_ost01_080 [165713.959174] Call Trace: [165713.974845] [] schedule_preempt_disabled+0x29/0x70 [165713.997422] [] __mutex_lock_slowpath+0xc7/0x1d0 [165714.019138] [] mutex_lock+0x1f/0x2f [165714.037774] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [165714.060235] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [165714.085353] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [165714.110587] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [165714.134368] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [165714.158693] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [165714.185562] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [165714.209509] [] ? default_wake_function+0x0/0x20 [165714.231446] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [165714.253654] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [165714.275809] [] kthread+0xcf/0xe0 [165714.293354] [] ? kthread+0x0/0xe0 [165714.311145] [] ret_from_fork+0x58/0x90 [165714.330334] [] ? kthread+0x0/0xe0 [165714.348058] [165714.354032] Pid: 222254, comm: ll_ost01_069 [165714.368879] Call Trace: [165714.383828] [] schedule_preempt_disabled+0x29/0x70 [165714.406022] [] __mutex_lock_slowpath+0xc7/0x1d0 [165714.427335] [] mutex_lock+0x1f/0x2f [165714.445216] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [165714.466829] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [165714.491284] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [165714.515719] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [165714.538705] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [165714.562236] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [165714.588328] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [165714.611515] [] ? default_wake_function+0x12/0x20 [165714.632973] [] ? __wake_up_common+0x58/0x90 [165714.653008] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [165714.674477] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [165714.695957] [] kthread+0xcf/0xe0 [165714.712848] [] ? kthread+0x0/0xe0 [165714.730011] [] ret_from_fork+0x58/0x90 [165714.748600] [] ? kthread+0x0/0xe0 [165714.765752] [165714.771448] Pid: 224652, comm: ll_ost01_092 [165714.786025] Call Trace: [165714.800562] [] schedule_preempt_disabled+0x29/0x70 [165714.822593] [] __mutex_lock_slowpath+0xc7/0x1d0 [165714.843763] [] mutex_lock+0x1f/0x2f [165714.861499] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [165714.882990] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [165714.907339] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [165714.931688] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [165714.954608] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [165714.978095] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [165715.004157] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [165715.027339] [] ? default_wake_function+0x12/0x20 [165715.048812] [] ? __wake_up_common+0x58/0x90 [165715.068876] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [165715.090364] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [165715.111827] [] kthread+0xcf/0xe0 [165715.128706] [] ? kthread+0x0/0xe0 [165715.145872] [] ret_from_fork+0x58/0x90 [165715.164458] [] ? kthread+0x0/0xe0 [165715.181611] [165715.187317] Pid: 29889, comm: ll_ost01_008 [165715.201612] Call Trace: [165715.216152] [] schedule_preempt_disabled+0x29/0x70 [165715.238176] [] __mutex_lock_slowpath+0xc7/0x1d0 [165715.259342] [] mutex_lock+0x1f/0x2f [165715.277100] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [165715.286692] kworker/u480:2: page allocation failure: order:8, mode:0x80d0 [165715.286693] CPU: 0 PID: 67636 Comm: kworker/u480:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165715.286694] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165715.286699] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165715.286700] 00000000000080d0 00000000987a9902 ffff88052cdf7810 ffffffff816a3e2d [165715.286701] ffff88052cdf78a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165715.286703] ffffffffffffff00 000080d000000000 ffff88052cdf7870 00000000987a9902 [165715.286703] Call Trace: [165715.286706] [] dump_stack+0x19/0x1b [165715.286708] [] warn_alloc_failed+0x110/0x180 [165715.286709] [] ? drain_pages+0xb0/0xb0 [165715.286710] [] __alloc_pages_slowpath+0x6b6/0x724 [165715.286712] [] __alloc_pages_nodemask+0x405/0x420 [165715.286714] [] dma_generic_alloc_coherent+0x8f/0x140 [165715.286715] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165715.286727] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165715.286732] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165715.286738] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165715.286742] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165715.286746] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165715.286751] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165715.286758] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165715.286760] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165715.286765] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165715.286769] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165715.286772] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165715.286773] [] process_one_work+0x17a/0x440 [165715.286775] [] worker_thread+0x126/0x3c0 [165715.286776] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165715.286777] [] kthread+0xcf/0xe0 [165715.286778] [] ? insert_kthread_work+0x40/0x40 [165715.286780] [] ret_from_fork+0x58/0x90 [165715.286781] [] ? insert_kthread_work+0x40/0x40 [165715.286781] Mem-Info: [165715.286786] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:103 inactive_file:83 isolated_file:0 unevictable:6774 dirty:0 writeback:9 unstable:0 slab_reclaimable:12113 slab_unreclaimable:560312 mapped:1544 shmem:0 pagetables:1412 bounce:0 free:39476 free_pcp:24 free_cma:0 [165715.286790] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165715.286791] lowmem_reserve[]: 0 2580 15620 15620 [165715.286795] Node 0 DMA32 free:59516kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:100kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:36kB mapped:328kB shmem:0kB slab_reclaimable:3132kB slab_unreclaimable:194188kB kernel_stack:1600kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165715.286796] lowmem_reserve[]: 0 0 13040 13040 [165715.286800] Node 0 Normal free:37120kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:48kB active_file:4kB inactive_file:164kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089652kB kernel_stack:14192kB pagetables:2160kB unstable:0kB bounce:0kB free_pcp:96kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165715.286801] lowmem_reserve[]: 0 0 0 0 [165715.286805] Node 1 Normal free:45688kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:36kB active_file:308kB inactive_file:292kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25912kB slab_unreclaimable:957408kB kernel_stack:16768kB pagetables:3272kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:991 all_unreclaimable? yes [165715.286806] lowmem_reserve[]: 0 0 0 0 [165715.286811] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165715.286816] Node 0 DMA32: 41*4kB (UEM) 1922*8kB (UEM) 1341*16kB (UEM) 500*32kB (UEM) 99*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59716kB [165715.286820] Node 0 Normal: 98*4kB (UEM) 145*8kB (EM) 113*16kB (EM) 241*32kB (UEM) 383*64kB (UEM) 12*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37120kB [165715.286824] Node 1 Normal: 558*4kB (EM) 387*8kB (EM) 210*16kB (EM) 358*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45936kB [165715.286825] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165715.286826] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165715.286826] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165715.286827] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165715.286827] 1793 total pagecache pages [165715.286828] 21 pages in swap cache [165715.286829] Swap cache stats: add 314075, delete 314054, find 93558/151038 [165715.286829] Free swap = 16257416kB [165715.286829] Total swap = 16319484kB [165715.286830] 8369063 pages RAM [165715.286830] 0 pages HighMem/MovableOnly [165715.286831] 241261 pages reserved [165715.287147] kworker/u480:2: page allocation failure: order:8, mode:0x80d0 [165715.287149] CPU: 16 PID: 67636 Comm: kworker/u480:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165715.287150] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165715.287154] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165715.287156] 00000000000080d0 00000000987a9902 ffff88052cdf77c0 ffffffff816a3e2d [165715.287158] ffff88052cdf7850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165715.287159] ffffffffffffff00 000080d000000000 ffff88052cdf7820 00000000987a9902 [165715.287159] Call Trace: [165715.287162] [] dump_stack+0x19/0x1b [165715.287164] [] warn_alloc_failed+0x110/0x180 [165715.287166] [] ? drain_pages+0xb0/0xb0 [165715.287169] [] __alloc_pages_slowpath+0x6b6/0x724 [165715.287171] [] __alloc_pages_nodemask+0x405/0x420 [165715.287172] [] alloc_pages_current+0x98/0x110 [165715.287174] [] __get_free_pages+0xe/0x40 [165715.287176] [] swiotlb_alloc_coherent+0x5e/0x150 [165715.287178] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165715.287186] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165715.287192] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165715.287197] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165715.287202] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165715.287206] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165715.287212] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165715.287219] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165715.287221] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165715.287225] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165715.287230] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165715.287233] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165715.287234] [] process_one_work+0x17a/0x440 [165715.287236] [] worker_thread+0x126/0x3c0 [165715.287238] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165715.287239] [] kthread+0xcf/0xe0 [165715.287240] [] ? insert_kthread_work+0x40/0x40 [165715.287242] [] ret_from_fork+0x58/0x90 [165715.287244] [] ? insert_kthread_work+0x40/0x40 [165715.287244] Mem-Info: [165715.287250] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:103 inactive_file:83 isolated_file:0 unevictable:6774 dirty:0 writeback:9 unstable:0 slab_reclaimable:12113 slab_unreclaimable:560312 mapped:1544 shmem:0 pagetables:1412 bounce:0 free:39476 free_pcp:57 free_cma:0 [165715.287255] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165715.287256] lowmem_reserve[]: 0 2580 15620 15620 [165715.287261] Node 0 DMA32 free:59516kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:0kB active_file:100kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:36kB mapped:328kB shmem:0kB slab_reclaimable:3132kB slab_unreclaimable:194188kB kernel_stack:1600kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165715.287262] lowmem_reserve[]: 0 0 13040 13040 [165715.287268] Node 0 Normal free:37120kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:48kB active_file:4kB inactive_file:164kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089652kB kernel_stack:14192kB pagetables:2160kB unstable:0kB bounce:0kB free_pcp:228kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165715.287269] lowmem_reserve[]: 0 0 0 0 [165715.287274] Node 1 Normal free:45688kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:36kB active_file:308kB inactive_file:292kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25912kB slab_unreclaimable:957408kB kernel_stack:16768kB pagetables:3272kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:991 all_unreclaimable? yes [165715.287275] lowmem_reserve[]: 0 0 0 0 [165715.287282] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165715.287288] Node 0 DMA32: 36*4kB (UEM) 1915*8kB (UEM) 1336*16kB (UEM) 502*32kB (UEM) 101*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59752kB [165715.287294] Node 0 Normal: 90*4kB (EM) 145*8kB (EM) 114*16kB (UEM) 234*32kB (UEM) 383*64kB (UEM) 12*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36880kB [165715.287300] Node 1 Normal: 558*4kB (EM) 387*8kB (EM) 210*16kB (EM) 358*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45936kB [165715.287301] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165715.287302] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165715.287303] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165715.287304] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165715.287304] 1793 total pagecache pages [165715.287305] 15 pages in swap cache [165715.287306] Swap cache stats: add 314075, delete 314060, find 93558/151038 [165715.287306] Free swap = 16257416kB [165715.287307] Total swap = 16319484kB [165715.287308] 8369063 pages RAM [165715.287308] 0 pages HighMem/MovableOnly [165715.287308] 241261 pages reserved [165719.257179] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [165719.282092] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [165719.307008] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [165719.330481] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [165719.354518] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [165719.381134] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [165719.404854] [] ? default_wake_function+0x12/0x20 [165719.426837] [] ? __wake_up_common+0x58/0x90 [165719.447393] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [165719.469372] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [165719.491306] [] kthread+0xcf/0xe0 [165719.508648] [] ? kthread+0x0/0xe0 [165719.526211] [] ret_from_fork+0x58/0x90 [165719.545207] [] ? kthread+0x0/0xe0 [165719.562742] [165719.568811] Pid: 150433, comm: ll_ost01_058 [165719.583734] Call Trace: [165719.598884] [] schedule_preempt_disabled+0x29/0x70 [165719.621192] [] __mutex_lock_slowpath+0xc7/0x1d0 [165719.642616] [] mutex_lock+0x1f/0x2f [165719.660614] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [165719.682329] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [165719.706887] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [165719.731426] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [165719.754515] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [165719.778153] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [165719.804370] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [165719.827699] [] ? default_wake_function+0x12/0x20 [165719.849286] [] ? __wake_up_common+0x58/0x90 [165719.869446] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [165719.891015] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [165719.912536] [] kthread+0xcf/0xe0 [165719.929457] [] ? kthread+0x0/0xe0 [165719.946637] [] ret_from_fork+0x58/0x90 [165719.965224] [] ? kthread+0x0/0xe0 [165719.982362] [165720.002817] systemd-journal invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [165720.031105] systemd-journal cpuset=/ mems_allowed=0-1 [165720.048806] CPU: 19 PID: 977 Comm: systemd-journal Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165720.088175] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165720.126268] ffff8800369b0fd0 00000000365a43d3 ffff880429dbf920 ffffffff816a3e2d [165720.151702] ffff880429dbf9b0 ffffffff8169f246 ffffffff810e93ac ffff88041729df30 [165720.177288] ffff88041729df48 0000000000000202 0000000000000202 ffff880429dbf9a0 [165720.202744] Call Trace: [165720.211965] [] dump_stack+0x19/0x1b [165720.230004] [] dump_header+0x90/0x229 [165720.248589] [] ? ktime_get_ts64+0x4c/0xf0 [165720.268422] [] ? delayacct_end+0x8f/0xb0 [165720.287935] [] oom_kill_process+0x254/0x3d0 [165720.308323] [] ? oom_unkillable_task+0xcd/0x120 [165720.329729] [] ? find_lock_task_mm+0x56/0xc0 [165720.350336] [] out_of_memory+0x4b6/0x4f0 [165720.369629] [] __alloc_pages_slowpath+0x5d6/0x724 [165720.391464] [] __alloc_pages_nodemask+0x405/0x420 [165720.413386] [] alloc_pages_current+0x98/0x110 [165720.434183] [] __page_cache_alloc+0x97/0xb0 [165720.454386] [] filemap_fault+0x170/0x410 [165720.473770] [] ext4_filemap_fault+0x36/0x50 [ext4] [165720.495986] [] __do_fault+0x52/0xe0 [165720.513823] [] do_read_fault.isra.44+0x4b/0x130 [165720.535205] [] handle_mm_fault+0x691/0xfa0 [165720.555166] [] ? change_protection+0x65/0xa0 [165720.575605] [] __do_page_fault+0x154/0x450 [165720.595565] [] do_page_fault+0x35/0x90 [165720.614266] [] page_fault+0x28/0x30 [165720.632178] Mem-Info: [165720.640643] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:123 inactive_file:110 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12105 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:1412 bounce:0 free:39317 free_pcp:155 free_cma:0 [165720.753861] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165720.893604] lowmem_reserve[]: 0 2580 15620 15620 [165720.909946] Node 0 DMA32 free:59284kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:32kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3132kB slab_unreclaimable:194188kB kernel_stack:1600kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:244kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4055 all_unreclaimable? yes [165721.061736] lowmem_reserve[]: 0 0 13040 13040 [165721.077269] Node 0 Normal free:36840kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:256kB inactive_file:44kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14192kB pagetables:2160kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1982 all_unreclaimable? yes [165721.232726] lowmem_reserve[]: 0 0 0 0 [165721.246127] Node 1 Normal free:45564kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16688kB pagetables:3272kB unstable:0kB bounce:0kB free_pcp:376kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1065 all_unreclaimable? yes [165721.402687] lowmem_reserve[]: 0 0 0 0 [165721.416019] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165721.462619] Node 0 DMA32: 31*4kB (UEM) 1879*8kB (UEM) 1332*16kB (UE) 500*32kB (UEM) 102*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59380kB [165721.513585] Node 0 Normal: 132*4kB (UEM) 147*8kB (UEM) 114*16kB (UEM) 227*32kB (UEM) 383*64kB (UEM) 12*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36840kB [165721.565496] Node 1 Normal: 435*4kB (UEM) 389*8kB (UEM) 217*16kB (UEM) 351*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45348kB [165721.617217] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165721.647279] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165721.676474] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165721.706514] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165721.735697] 1847 total pagecache pages [165721.749464] 3 pages in swap cache [165721.761758] Swap cache stats: add 314188, delete 314185, find 93588/151100 [165721.785798] Free swap = 16257408kB [165721.798674] Total swap = 16319484kB [165721.811559] 8369063 pages RAM [165721.822715] 0 pages HighMem/MovableOnly [165721.836735] 241261 pages reserved [165721.849027] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165721.876219] [ 977] 0 977 20596 231 41 95 0 systemd-journal [165721.906563] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165721.934610] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165721.964379] [ 1046] 0 1046 1588718 3712 132 0 -1000 multipathd [165721.993253] [ 2373] 0 2373 13863 117 27 116 -1000 auditd [165722.020975] [ 2428] 0 2428 63244 296 58 591 0 rsyslogd [165722.049249] [ 2429] 0 2429 64920 323 78 278 0 sssd [165722.076351] [ 2434] 0 2434 67755 206 45 176 0 gssproxy [165722.104600] [ 2435] 0 2435 5461 184 15 148 0 irqbalance [165722.133382] [ 2444] 81 2444 9196 203 21 146 -900 dbus-daemon [165722.162434] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165722.190044] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165722.218195] [ 2771] 998 2771 29553 242 29 125 0 chronyd [165722.246046] [ 2798] 0 2798 93411 317 125 648 0 sssd_be [165722.273881] [ 2828] 0 2828 66338 284 85 245 0 sssd_nss [165722.301990] [ 2829] 0 2829 61252 282 75 230 0 sssd_pam [165722.330074] [ 2830] 0 2830 59613 260 71 246 0 sssd_ssh [165722.358130] [ 2831] 0 2831 68163 257 86 336 0 sssd_pac [165722.386156] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165722.413588] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165722.440448] [ 3703] 996 3703 24211 165 21 134 0 munged [165722.467860] [ 3875] 0 3875 22907 176 44 276 0 master [165722.495245] [ 3908] 89 3908 25491 217 47 276 0 qmgr [165722.522037] [ 3915] 0 3915 31559 192 18 170 0 crond [165722.549085] [ 3919] 0 3919 168252 207 70 405 0 automount [165722.577260] [ 3951] 0 3951 6464 155 19 57 0 atd [165722.603725] [ 3970] 0 3970 27511 156 12 32 0 agetty [165722.631018] [ 3971] 0 3971 27511 160 10 31 0 agetty [165722.658336] [67360] 89 67360 25474 215 47 263 0 pickup [165722.685581] [67627] 0 67627 40829 124 33 217 0 crond [165722.712516] [67635] 0 67635 90 18 4 9 0 systemd-cgroups [165722.742272] [67641] 0 67641 79 0 4 9 0 systemd-cgroups [165722.772022] Out of memory: Kill process 2798 (sssd_be) score 0 or sacrifice child [165722.797472] Killed process 2798 (sssd_be) total-vm:373644kB, anon-rss:0kB, file-rss:1268kB, shmem-rss:0kB [165722.860021] crond invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165722.885402] crond cpuset=/ mems_allowed=0-1 [165722.900221] CPU: 20 PID: 67627 Comm: crond Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165722.936506] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165722.974580] ffff880063a21fa0 00000000be59da59 ffff8802eb513988 ffffffff816a3e2d [165723.000132] ffff8802eb513a18 ffffffff8169f246 ffffffff810e93ac ffff8803dbf69130 [165723.025685] ffff8803dbf69148 0000000000000206 0000000000000206 ffff8802eb513a08 [165723.050983] Call Trace: [165723.060152] [] dump_stack+0x19/0x1b [165723.078208] [] dump_header+0x90/0x229 [165723.096836] [] ? ktime_get_ts64+0x4c/0xf0 [165723.116559] [] ? delayacct_end+0x8f/0xb0 [165723.136058] [] oom_kill_process+0x254/0x3d0 [165723.156373] [] ? oom_unkillable_task+0xcd/0x120 [165723.177807] [] ? find_lock_task_mm+0x56/0xc0 [165723.198416] [] out_of_memory+0x4b6/0x4f0 [165723.217867] [] __alloc_pages_slowpath+0x5d6/0x724 [165723.239905] [] __alloc_pages_nodemask+0x405/0x420 [165723.261940] [] alloc_pages_vma+0xb5/0x200 [165723.281669] [] ? end_swap_bio_write+0x80/0x80 [165723.302419] [] read_swap_cache_async+0xed/0x160 [165723.323829] [] ? mem_cgroup_update_page_stat+0x16/0x50 [165723.347296] [] swapin_readahead+0xa8/0x110 [165723.367229] [] handle_mm_fault+0xadb/0xfa0 [165723.387084] [] __do_page_fault+0x154/0x450 [165723.406988] [] do_page_fault+0x35/0x90 [165723.425787] [] page_fault+0x28/0x30 [165723.443662] Mem-Info: [165723.452156] active_anon:0 inactive_anon:9 isolated_anon:0 active_file:181 inactive_file:57 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12105 slab_unreclaimable:560306 mapped:1548 shmem:0 pagetables:1287 bounce:0 free:39305 free_pcp:32 free_cma:0 [165723.564941] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165723.704704] lowmem_reserve[]: 0 2580 15620 15620 [165723.721289] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:28kB inactive_file:4kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3132kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:160kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3904 all_unreclaimable? yes [165723.872731] lowmem_reserve[]: 0 0 13040 13040 [165723.888674] Node 0 Normal free:36840kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:12kB active_file:460kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1536kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14192kB pagetables:1716kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1500 all_unreclaimable? yes [165724.045921] lowmem_reserve[]: 0 0 0 0 [165724.059604] Node 1 Normal free:45320kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:20kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16640kB pagetables:3272kB unstable:0kB bounce:0kB free_pcp:136kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5244 all_unreclaimable? yes [165724.218493] lowmem_reserve[]: 0 0 0 0 [165724.232258] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165724.279804] Node 0 DMA32: 33*4kB (UEM) 1865*8kB (UEM) 1332*16kB (UEM) 502*32kB (UEM) 103*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59404kB [165724.331963] Node 0 Normal: 129*4kB (UEM) 145*8kB (UEM) 112*16kB (UEM) 227*32kB (UEM) 385*64kB (UEM) 13*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37036kB [165724.384757] Node 1 Normal: 559*4kB (UEM) 388*8kB (UEM) 210*16kB (EM) 346*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45564kB [165724.436660] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165724.466711] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165724.495902] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165724.526385] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165724.556019] 1779 total pagecache pages [165724.570203] 0 pages in swap cache [165724.582945] Swap cache stats: add 314254, delete 314254, find 93603/151128 [165724.607423] Free swap = 16259896kB [165724.620765] Total swap = 16319484kB [165724.634086] 8369063 pages RAM [165724.645699] 0 pages HighMem/MovableOnly [165724.660170] 241261 pages reserved [165724.672903] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165724.700571] [ 977] 0 977 20596 226 41 95 0 systemd-journal [165724.731401] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165724.759918] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165724.790157] [ 1046] 0 1046 1588718 3709 132 0 -1000 multipathd [165724.819542] [ 2373] 0 2373 13863 116 27 116 -1000 auditd [165724.847728] [ 2428] 0 2428 63244 296 58 594 0 rsyslogd [165724.876188] [ 2429] 0 2429 64920 323 78 278 0 sssd [165724.903481] [ 2434] 0 2434 67755 206 45 177 0 gssproxy [165724.931893] [ 2435] 0 2435 5461 183 15 150 0 irqbalance [165724.960849] [ 2444] 81 2444 9196 203 21 146 -900 dbus-daemon [165724.990094] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165725.017799] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165725.046369] [ 2771] 998 2771 29553 242 29 125 0 chronyd [165725.074359] [ 2828] 0 2828 66338 284 85 245 0 sssd_nss [165725.102875] [ 2829] 0 2829 61252 282 75 230 0 sssd_pam [165725.131336] [ 2830] 0 2830 59613 260 71 247 0 sssd_ssh [165725.159531] [ 2831] 0 2831 68163 256 86 337 0 sssd_pac [165725.187944] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165725.215747] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165725.242942] [ 3703] 996 3703 24211 165 21 134 0 munged [165725.270453] [ 3875] 0 3875 22907 175 44 277 0 master [165725.298165] [ 3908] 89 3908 25491 217 47 277 0 qmgr [165725.325311] [ 3915] 0 3915 31559 191 18 171 0 crond [165725.352476] [ 3919] 0 3919 168252 207 70 406 0 automount [165725.380744] [ 3951] 0 3951 6464 155 19 57 0 atd [165725.407241] [ 3970] 0 3970 27511 156 12 32 0 agetty [165725.434860] [ 3971] 0 3971 27511 160 10 31 0 agetty [165725.462485] [67360] 89 67360 25474 215 47 263 0 pickup [165725.490043] [67627] 0 67627 40829 123 33 217 0 crond [165725.517269] [67635] 0 67635 90 16 4 9 0 systemd-cgroups [165725.547320] [67641] 0 67641 79 0 4 13 0 systemd-cgroups [165725.577335] Out of memory: Kill process 2428 (rsyslogd) score 0 or sacrifice child [165725.603320] Killed process 2428 (rsyslogd) total-vm:252976kB, anon-rss:0kB, file-rss:1184kB, shmem-rss:0kB [165725.636784] rsyslogd: page allocation failure: order:0, mode:0x201da [165725.658640] CPU: 16 PID: 2428 Comm: rsyslogd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165725.695271] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165725.733071] 00000000000201da 000000007a2c64d3 ffff88082cc2fa30 ffffffff816a3e2d [165725.758363] ffff88082cc2fac0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165725.783894] ffffffffffffffff 000201da00000000 ffff88082cc2fac0 000000007a2c64d3 [165725.809454] Call Trace: [165725.818628] [] dump_stack+0x19/0x1b [165725.836677] [] warn_alloc_failed+0x110/0x180 [165725.857295] [] ? drain_pages+0xb0/0xb0 [165725.876185] [] ? wake_up_atomic_t+0x30/0x30 [165725.896517] [] __alloc_pages_slowpath+0x6b6/0x724 [165725.918585] [] __alloc_pages_nodemask+0x405/0x420 [165725.940656] [] alloc_pages_current+0x98/0x110 [165725.961313] [] __page_cache_alloc+0x97/0xb0 [165725.981387] [] filemap_fault+0x170/0x410 [165726.000864] [] ext4_filemap_fault+0x36/0x50 [ext4] [165726.023203] [] __do_fault+0x52/0xe0 [165726.041242] [] do_read_fault.isra.44+0x4b/0x130 [165726.062715] [] handle_mm_fault+0x691/0xfa0 [165726.082752] [] __do_page_fault+0x154/0x450 [165726.102795] [] do_page_fault+0x35/0x90 [165726.121689] [] page_fault+0x28/0x30 [165726.139716] Mem-Info: [165726.148306] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:157 inactive_file:56 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12105 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:1287 bounce:0 free:39384 free_pcp:93 free_cma:0 [165726.261618] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165726.401276] lowmem_reserve[]: 0 2580 15620 15620 [165726.417589] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:24kB inactive_file:8kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3132kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:160kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:146 all_unreclaimable? yes [165726.568404] lowmem_reserve[]: 0 0 13040 13040 [165726.584262] Node 0 Normal free:36788kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:336kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1536kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14192kB pagetables:1716kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:836 all_unreclaimable? yes [165726.740676] lowmem_reserve[]: 0 0 0 0 [165726.754360] Node 1 Normal free:45436kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16640kB pagetables:3272kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1166 all_unreclaimable? yes [165726.912442] lowmem_reserve[]: 0 0 0 0 [165726.926158] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165726.973564] Node 0 DMA32: 31*4kB (EM) 1878*8kB (UEM) 1332*16kB (UEM) 502*32kB (UEM) 103*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59500kB [165727.025345] Node 0 Normal: 130*4kB (UEM) 145*8kB (UEM) 112*16kB (UEM) 227*32kB (UEM) 385*64kB (UEM) 13*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37040kB [165727.078122] Node 1 Normal: 558*4kB (EM) 388*8kB (UEM) 210*16kB (EM) 346*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45560kB [165727.130107] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165727.160630] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165727.190225] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165727.220744] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165727.250350] 1779 total pagecache pages [165727.264512] 2 pages in swap cache [165727.277185] Swap cache stats: add 314274, delete 314272, find 93607/151138 [165727.301612] Free swap = 16259896kB [165727.314977] Total swap = 16319484kB [165727.328319] 8369063 pages RAM [165727.339969] 0 pages HighMem/MovableOnly [165727.354422] 241261 pages reserved [165727.380355] ll_ost_io01_104 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [165727.409257] ll_ost_io01_104 cpuset=/ mems_allowed=1 [165727.427188] CPU: 10 PID: 226541 Comm: ll_ost_io01_104 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165727.468159] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165727.507103] ffff88081d3a8fd0 00000000c8832be1 ffff8807947af998 ffffffff816a3e2d [165727.533311] ffff8807947afa28 ffffffff8169f246 ffffffff810e93ac ffff8807ebda5de0 [165727.559583] ffff8807ebda5df8 0000000000000206 0000000000000206 ffff8807947afa18 [165727.586044] Call Trace: [165727.595951] [] dump_stack+0x19/0x1b [165727.614804] [] dump_header+0x90/0x229 [165727.634211] [] ? ktime_get_ts64+0x4c/0xf0 [165727.654706] [] ? delayacct_end+0x8f/0xb0 [165727.674663] [] oom_kill_process+0x254/0x3d0 [165727.695427] [] ? oom_unkillable_task+0xcd/0x120 [165727.717687] [] ? find_lock_task_mm+0x56/0xc0 [165727.739052] [] out_of_memory+0x4b6/0x4f0 [165727.759228] [] __alloc_pages_slowpath+0x5d6/0x724 [165727.781958] [] __alloc_pages_nodemask+0x405/0x420 [165727.804614] [] __vmalloc_node_range+0x12f/0x270 [165727.826208] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165727.850027] [] vzalloc_node+0x47/0x50 [165727.868709] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165727.892924] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165727.916154] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [165727.940311] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [165727.962582] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [165727.988217] [] kthread+0xcf/0xe0 [165728.005516] [] ? insert_kthread_work+0x40/0x40 [165728.026828] [] ret_from_fork+0x58/0x90 [165728.045936] [] ? insert_kthread_work+0x40/0x40 [165728.067474] Mem-Info: [165728.075967] active_anon:0 inactive_anon:11 isolated_anon:0 active_file:161 inactive_file:44 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12105 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:1287 bounce:0 free:39353 free_pcp:0 free_cma:0 [165728.190536] Node 1 Normal free:45560kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16640kB pagetables:3272kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1036 all_unreclaimable? yes [165728.345690] lowmem_reserve[]: 0 0 0 0 [165728.358835] Node 1 Normal: 572*4kB (UEM) 391*8kB (UEM) 212*16kB (UEM) 347*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45704kB [165728.410604] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165728.440685] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165728.469768] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165728.499655] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165728.528804] 1815 total pagecache pages [165728.542472] 12 pages in swap cache [165728.554970] Swap cache stats: add 314364, delete 314352, find 93643/151187 [165728.578889] Free swap = 16262212kB [165728.591662] Total swap = 16319484kB [165728.604193] 8369063 pages RAM [165728.615218] 0 pages HighMem/MovableOnly [165728.629097] 241261 pages reserved [165728.641266] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165728.668354] [ 977] 0 977 20596 225 41 97 0 systemd-journal [165728.698589] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165728.726531] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165728.756201] [ 1046] 0 1046 1588718 3700 132 0 -1000 multipathd [165728.785013] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165728.812676] [ 2429] 0 2429 64920 323 78 279 0 sssd [165728.839741] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165728.867937] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165728.896743] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165728.925742] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165728.953400] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165728.981611] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.009542] [ 2828] 0 2828 66338 284 85 246 0 sssd_nss [165729.037641] [ 2829] 0 2829 61252 282 75 231 0 sssd_pam [165729.065853] [ 2830] 0 2830 59613 269 71 220 0 sssd_ssh [165729.093940] [ 2831] 0 2831 68163 256 86 338 0 sssd_pac [165729.122041] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.149501] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.176403] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.204003] [ 3875] 0 3875 22907 174 44 280 0 master [165729.231594] [ 3908] 89 3908 25491 216 47 278 0 qmgr [165729.258603] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.285902] [ 3919] 0 3919 168252 207 70 408 0 automount [165729.314195] [ 3951] 0 3951 6464 154 19 58 0 atd [165729.340740] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.368107] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.395467] [67360] 89 67360 25474 215 47 264 0 pickup [165729.422754] [67627] 0 67627 40829 122 33 218 0 crond [165729.449716] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.479542] [67641] 0 67641 79 0 4 19 0 systemd-cgroups [165729.509323] [67646] 0 67646 1 0 1 1 0 systemd-cgroups [165729.539069] Out of memory: Kill process 3919 (automount) score 0 or sacrifice child [165729.565107] Killed process 3919 (automount) total-vm:673008kB, anon-rss:0kB, file-rss:828kB, shmem-rss:0kB [165729.732385] crond invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165729.732387] crond cpuset=/ mems_allowed=0-1 [165729.732389] CPU: 13 PID: 67627 Comm: crond Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.732390] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.732393] ffff880063a21fa0 00000000be59da59 ffff8802eb513988 ffffffff816a3e2d [165729.732394] ffff8802eb513a18 ffffffff8169f246 ffffffff810e93ac ffff8803dbf69130 [165729.732396] ffff8803dbf69148 0000000000000202 0000000000000202 ffff8802eb513a08 [165729.732396] Call Trace: [165729.732401] [] dump_stack+0x19/0x1b [165729.732402] [] dump_header+0x90/0x229 [165729.732404] [] ? ktime_get_ts64+0x4c/0xf0 [165729.732406] [] ? delayacct_end+0x8f/0xb0 [165729.732409] [] oom_kill_process+0x254/0x3d0 [165729.732411] [] ? oom_unkillable_task+0xcd/0x120 [165729.732413] [] ? find_lock_task_mm+0x56/0xc0 [165729.732415] [] out_of_memory+0x4b6/0x4f0 [165729.732417] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.732419] [] __alloc_pages_nodemask+0x405/0x420 [165729.732422] [] alloc_pages_vma+0xb5/0x200 [165729.732425] [] read_swap_cache_async+0xed/0x160 [165729.732427] [] swapin_readahead+0xa8/0x110 [165729.732429] [] handle_mm_fault+0xadb/0xfa0 [165729.732433] [] __do_page_fault+0x154/0x450 [165729.732436] [] do_page_fault+0x35/0x90 [165729.732438] [] page_fault+0x28/0x30 [165729.732439] Mem-Info: [165729.732445] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:413 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:1208 bounce:0 free:39349 free_pcp:325 free_cma:0 [165729.732452] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.732454] lowmem_reserve[]: 0 2580 15620 15620 [165729.732460] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6940 all_unreclaimable? yes [165729.732462] lowmem_reserve[]: 0 0 13040 13040 [165729.732468] Node 0 Normal free:36676kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1400kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:1620kB unstable:0kB bounce:0kB free_pcp:544kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:23321 all_unreclaimable? yes [165729.732469] lowmem_reserve[]: 0 0 0 0 [165729.732475] Node 1 Normal free:45704kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:3148kB unstable:0kB bounce:0kB free_pcp:652kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29827 all_unreclaimable? yes [165729.732476] lowmem_reserve[]: 0 0 0 0 [165729.732485] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.732492] Node 0 DMA32: 33*4kB (UEM) 1867*8kB (UEM) 1333*16kB (UEM) 504*32kB (UEM) 102*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59436kB [165729.732499] Node 0 Normal: 44*4kB (E) 139*8kB (UEM) 110*16kB (EM) 215*32kB (UEM) 385*64kB (UEM) 13*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36232kB [165729.732507] Node 1 Normal: 448*4kB (UEM) 391*8kB (UEM) 214*16kB (UEM) 347*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45240kB [165729.732508] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.732509] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.732510] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.732511] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.732512] 1785 total pagecache pages [165729.732513] 3 pages in swap cache [165729.732514] Swap cache stats: add 314407, delete 314404, find 93654/151209 [165729.732515] Free swap = 16263672kB [165729.732515] Total swap = 16319484kB [165729.732516] 8369063 pages RAM [165729.732516] 0 pages HighMem/MovableOnly [165729.732517] 241261 pages reserved [165729.732517] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.732525] [ 977] 0 977 20596 225 41 97 0 systemd-journal [165729.732527] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.732529] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.732531] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.732540] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.732542] [ 2429] 0 2429 64920 323 78 279 0 sssd [165729.732543] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.732545] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.732547] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.732549] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.732550] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.732552] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.732554] [ 2828] 0 2828 66338 284 85 246 0 sssd_nss [165729.732555] [ 2829] 0 2829 61252 282 75 231 0 sssd_pam [165729.732557] [ 2830] 0 2830 59613 269 71 220 0 sssd_ssh [165729.732559] [ 2831] 0 2831 68163 256 86 338 0 sssd_pac [165729.732561] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.732562] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.732564] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.732566] [ 3875] 0 3875 22907 174 44 280 0 master [165729.732567] [ 3908] 89 3908 25491 216 47 278 0 qmgr [165729.732568] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.732570] [ 3951] 0 3951 6464 154 19 58 0 atd [165729.732572] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.732573] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.732600] [67360] 89 67360 25474 215 47 264 0 pickup [165729.732602] [67627] 0 67627 40829 122 33 218 0 crond [165729.732604] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.732606] [67641] 0 67641 79 0 4 19 0 systemd-cgroups [165729.732608] [67646] 0 67646 1 0 1 1 0 systemd-cgroups [165729.732609] Out of memory: Kill process 2429 (sssd) score 0 or sacrifice child [165729.732611] Killed process 2831 (sssd_pac) total-vm:272652kB, anon-rss:0kB, file-rss:1024kB, shmem-rss:0kB [165729.734805] auditd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=-1000 [165729.734806] auditd cpuset=/ mems_allowed=0-1 [165729.734807] CPU: 2 PID: 2373 Comm: auditd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.734808] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.734810] ffff88082c1caf70 00000000689d2a54 ffff88082ae7f920 ffffffff816a3e2d [165729.734811] ffff88082ae7f9b0 ffffffff8169f246 ffffffff810e93ac ffff8808162f74b0 [165729.734812] ffff8808162f74c8 0000000000000202 0000000000000202 ffff88082ae7f9a0 [165729.734812] Call Trace: [165729.734815] [] dump_stack+0x19/0x1b [165729.734816] [] dump_header+0x90/0x229 [165729.734818] [] ? ktime_get_ts64+0x4c/0xf0 [165729.734819] [] ? delayacct_end+0x8f/0xb0 [165729.734821] [] oom_kill_process+0x254/0x3d0 [165729.734822] [] ? oom_unkillable_task+0xcd/0x120 [165729.734824] [] ? find_lock_task_mm+0x56/0xc0 [165729.734825] [] out_of_memory+0x4b6/0x4f0 [165729.734827] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.734828] [] __alloc_pages_nodemask+0x405/0x420 [165729.734830] [] alloc_pages_current+0x98/0x110 [165729.734831] [] __page_cache_alloc+0x97/0xb0 [165729.734833] [] filemap_fault+0x170/0x410 [165729.734843] [] ext4_filemap_fault+0x36/0x50 [ext4] [165729.734845] [] __do_fault+0x52/0xe0 [165729.734846] [] do_read_fault.isra.44+0x4b/0x130 [165729.734847] [] handle_mm_fault+0x691/0xfa0 [165729.734849] [] ? change_protection+0x65/0xa0 [165729.734851] [] __do_page_fault+0x154/0x450 [165729.734852] [] do_page_fault+0x35/0x90 [165729.734854] [] page_fault+0x28/0x30 [165729.734854] Mem-Info: [165729.734859] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:413 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:1099 bounce:0 free:39349 free_pcp:479 free_cma:0 [165729.734864] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.734865] lowmem_reserve[]: 0 2580 15620 15620 [165729.734869] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6956 all_unreclaimable? yes [165729.734870] lowmem_reserve[]: 0 0 13040 13040 [165729.734873] Node 0 Normal free:36676kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1400kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:1184kB unstable:0kB bounce:0kB free_pcp:1160kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:23326 all_unreclaimable? yes [165729.734874] lowmem_reserve[]: 0 0 0 0 [165729.734878] Node 1 Normal free:45704kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:3148kB unstable:0kB bounce:0kB free_pcp:652kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29827 all_unreclaimable? yes [165729.734879] lowmem_reserve[]: 0 0 0 0 [165729.734884] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.734888] Node 0 DMA32: 33*4kB (UEM) 1867*8kB (UEM) 1333*16kB (UEM) 504*32kB (UEM) 102*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59436kB [165729.734893] Node 0 Normal: 46*4kB (UEM) 123*8kB (UEM) 110*16kB (EM) 211*32kB (UEM) 385*64kB (UEM) 13*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 35984kB [165729.734897] Node 1 Normal: 448*4kB (UEM) 391*8kB (UEM) 214*16kB (UEM) 347*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45240kB [165729.734898] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.734898] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.734899] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.734899] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.734900] 1785 total pagecache pages [165729.734900] 3 pages in swap cache [165729.734901] Swap cache stats: add 314410, delete 314407, find 93655/151211 [165729.734901] Free swap = 16264944kB [165729.734901] Total swap = 16319484kB [165729.734902] 8369063 pages RAM [165729.734902] 0 pages HighMem/MovableOnly [165729.734903] 241261 pages reserved [165729.734903] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.734910] [ 977] 0 977 20596 225 41 97 0 systemd-journal [165729.734911] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.734913] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.734914] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.734922] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.734923] [ 2429] 0 2429 64920 323 78 279 0 sssd [165729.734924] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.734925] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.734926] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.734927] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.734928] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.734930] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.734931] [ 2828] 0 2828 66338 284 85 246 0 sssd_nss [165729.734932] [ 2829] 0 2829 61252 282 75 231 0 sssd_pam [165729.734933] [ 2830] 0 2830 59613 269 71 220 0 sssd_ssh [165729.734934] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.734935] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.734936] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.734937] [ 3875] 0 3875 22907 174 44 280 0 master [165729.734938] [ 3908] 89 3908 25491 216 47 278 0 qmgr [165729.734940] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.734941] [ 3951] 0 3951 6464 154 19 58 0 atd [165729.734942] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.734943] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.734968] [67360] 89 67360 25474 215 47 264 0 pickup [165729.734970] [67627] 0 67627 40829 122 33 218 0 crond [165729.734971] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.734972] [67641] 0 67641 79 0 4 19 0 systemd-cgroups [165729.734973] [67646] 0 67646 1 0 1 1 0 systemd-cgroups [165729.734974] Out of memory: Kill process 2429 (sssd) score 0 or sacrifice child [165729.734976] Killed process 2828 (sssd_nss) total-vm:265352kB, anon-rss:0kB, file-rss:1136kB, shmem-rss:0kB [165729.758129] master invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165729.758130] master cpuset=/ mems_allowed=0-1 [165729.758132] CPU: 23 PID: 3875 Comm: master Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.758132] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.758134] ffff88042bb95ee0 00000000d54cb491 ffff880414e57988 ffffffff816a3e2d [165729.758135] ffff880414e57a18 ffffffff8169f246 ffffffff810e93ac ffff880409e21210 [165729.758136] ffff880409e21228 0000000000000202 0000000000000202 ffff880414e57a08 [165729.758137] Call Trace: [165729.758139] [] dump_stack+0x19/0x1b [165729.758140] [] dump_header+0x90/0x229 [165729.758142] [] ? ktime_get_ts64+0x4c/0xf0 [165729.758143] [] ? delayacct_end+0x8f/0xb0 [165729.758145] [] oom_kill_process+0x254/0x3d0 [165729.758147] [] ? oom_unkillable_task+0xcd/0x120 [165729.758148] [] ? find_lock_task_mm+0x56/0xc0 [165729.758149] [] out_of_memory+0x4b6/0x4f0 [165729.758151] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.758152] [] __alloc_pages_nodemask+0x405/0x420 [165729.758154] [] alloc_pages_vma+0xb5/0x200 [165729.758155] [] read_swap_cache_async+0xed/0x160 [165729.758157] [] swapin_readahead+0xa8/0x110 [165729.758158] [] handle_mm_fault+0xadb/0xfa0 [165729.758160] [] __do_page_fault+0x154/0x450 [165729.758161] [] do_page_fault+0x35/0x90 [165729.758163] [] page_fault+0x28/0x30 [165729.758163] Mem-Info: [165729.758167] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:413 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:1099 bounce:0 free:39349 free_pcp:595 free_cma:0 [165729.758171] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.758172] lowmem_reserve[]: 0 2580 15620 15620 [165729.758176] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7212 all_unreclaimable? yes [165729.758177] lowmem_reserve[]: 0 0 13040 13040 [165729.758181] Node 0 Normal free:36676kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1400kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:1184kB unstable:0kB bounce:0kB free_pcp:1256kB local_pcp:608kB free_cma:0kB writeback_tmp:0kB pages_scanned:23329 all_unreclaimable? yes [165729.758182] lowmem_reserve[]: 0 0 0 0 [165729.758185] Node 1 Normal free:45704kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:3148kB unstable:0kB bounce:0kB free_pcp:1020kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29827 all_unreclaimable? yes [165729.758186] lowmem_reserve[]: 0 0 0 0 [165729.758191] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.758195] Node 0 DMA32: 33*4kB (UEM) 1867*8kB (UEM) 1333*16kB (UEM) 504*32kB (UEM) 102*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59436kB [165729.758200] Node 0 Normal: 45*4kB (EM) 122*8kB (EM) 111*16kB (UEM) 207*32kB (UEM) 385*64kB (UEM) 13*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 35860kB [165729.758204] Node 1 Normal: 448*4kB (UEM) 391*8kB (UEM) 214*16kB (UEM) 347*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45240kB [165729.758205] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.758205] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.758206] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.758206] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.758207] 1785 total pagecache pages [165729.758207] 3 pages in swap cache [165729.758208] Swap cache stats: add 314412, delete 314409, find 93656/151213 [165729.758208] Free swap = 16265912kB [165729.758209] Total swap = 16319484kB [165729.758209] 8369063 pages RAM [165729.758210] 0 pages HighMem/MovableOnly [165729.758210] 241261 pages reserved [165729.758210] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.758217] [ 977] 0 977 20596 225 41 97 0 systemd-journal [165729.758218] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.758219] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.758220] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.758227] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.758228] [ 2429] 0 2429 64920 323 78 279 0 sssd [165729.758229] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.758230] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.758231] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.758232] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.758233] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.758234] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.758235] [ 2829] 0 2829 61252 282 75 231 0 sssd_pam [165729.758236] [ 2830] 0 2830 59613 269 71 220 0 sssd_ssh [165729.758238] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.758239] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.758240] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.758241] [ 3875] 0 3875 22907 174 44 280 0 master [165729.758241] [ 3908] 89 3908 25491 216 47 278 0 qmgr [165729.758243] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.758244] [ 3951] 0 3951 6464 154 19 58 0 atd [165729.758244] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.758245] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.758270] [67360] 89 67360 25474 215 47 264 0 pickup [165729.758271] [67627] 0 67627 40829 122 33 218 0 crond [165729.758272] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.758274] [67641] 0 67641 79 0 4 19 0 systemd-cgroups [165729.758275] [67646] 0 67646 1 0 1 1 0 systemd-cgroups [165729.758276] Out of memory: Kill process 2429 (sssd) score 0 or sacrifice child [165729.758277] Killed process 2829 (sssd_pam) total-vm:245008kB, anon-rss:0kB, file-rss:1128kB, shmem-rss:0kB [165729.760940] qmgr invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [165729.760941] qmgr cpuset=/ mems_allowed=0-1 [165729.760943] CPU: 6 PID: 3908 Comm: qmgr Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.760944] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.760946] ffff88081ddb8000 0000000054c8457a ffff8808217bb920 ffffffff816a3e2d [165729.760947] ffff8808217bb9b0 ffffffff8169f246 ffffffff810e93ac ffff8807fde3a910 [165729.760949] ffff8807fde3a928 0000000000000202 0000000000000202 ffff8808217bb9a0 [165729.760949] Call Trace: [165729.760952] [] dump_stack+0x19/0x1b [165729.760954] [] dump_header+0x90/0x229 [165729.760956] [] ? ktime_get_ts64+0x4c/0xf0 [165729.760957] [] ? delayacct_end+0x8f/0xb0 [165729.760959] [] oom_kill_process+0x254/0x3d0 [165729.760961] [] ? oom_unkillable_task+0xcd/0x120 [165729.760963] [] ? find_lock_task_mm+0x56/0xc0 [165729.760965] [] out_of_memory+0x4b6/0x4f0 [165729.760966] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.760968] [] __alloc_pages_nodemask+0x405/0x420 [165729.760970] [] alloc_pages_current+0x98/0x110 [165729.760972] [] __page_cache_alloc+0x97/0xb0 [165729.760974] [] filemap_fault+0x170/0x410 [165729.760985] [] ext4_filemap_fault+0x36/0x50 [ext4] [165729.760987] [] __do_fault+0x52/0xe0 [165729.760988] [] do_read_fault.isra.44+0x4b/0x130 [165729.760990] [] handle_mm_fault+0x691/0xfa0 [165729.760992] [] ? change_protection+0x65/0xa0 [165729.760994] [] __do_page_fault+0x154/0x450 [165729.760996] [] do_page_fault+0x35/0x90 [165729.760998] [] page_fault+0x28/0x30 [165729.760999] Mem-Info: [165729.761008] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:413 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:990 bounce:0 free:39349 free_pcp:676 free_cma:0 [165729.761013] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.761015] lowmem_reserve[]: 0 2580 15620 15620 [165729.761020] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7236 all_unreclaimable? yes [165729.761022] lowmem_reserve[]: 0 0 13040 13040 [165729.761026] Node 0 Normal free:36676kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1400kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:748kB unstable:0kB bounce:0kB free_pcp:1580kB local_pcp:24kB free_cma:0kB writeback_tmp:0kB pages_scanned:23332 all_unreclaimable? yes [165729.761027] lowmem_reserve[]: 0 0 0 0 [165729.761032] Node 1 Normal free:45704kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:3148kB unstable:0kB bounce:0kB free_pcp:1020kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29827 all_unreclaimable? yes [165729.761033] lowmem_reserve[]: 0 0 0 0 [165729.761041] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.761047] Node 0 DMA32: 33*4kB (UEM) 1867*8kB (UEM) 1333*16kB (UEM) 504*32kB (UEM) 102*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59436kB [165729.761053] Node 0 Normal: 45*4kB (EM) 122*8kB (EM) 111*16kB (UEM) 207*32kB (UEM) 385*64kB (UEM) 13*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 35860kB [165729.761059] Node 1 Normal: 448*4kB (UEM) 391*8kB (UEM) 214*16kB (UEM) 347*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45240kB [165729.761060] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.761061] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.761062] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.761062] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.761063] 1785 total pagecache pages [165729.761063] 3 pages in swap cache [165729.761064] Swap cache stats: add 314415, delete 314412, find 93657/151215 [165729.761065] Free swap = 16266828kB [165729.761065] Total swap = 16319484kB [165729.761066] 8369063 pages RAM [165729.761066] 0 pages HighMem/MovableOnly [165729.761067] 241261 pages reserved [165729.761067] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.761074] [ 977] 0 977 20596 225 41 97 0 systemd-journal [165729.761076] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.761077] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.761079] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.761086] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.761088] [ 2429] 0 2429 64920 323 78 279 0 sssd [165729.761089] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.761091] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.761092] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.761094] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.761095] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.761097] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.761099] [ 2830] 0 2830 59613 269 71 220 0 sssd_ssh [165729.761101] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.761102] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.761104] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.761105] [ 3875] 0 3875 22907 174 44 280 0 master [165729.761107] [ 3908] 89 3908 25491 216 47 278 0 qmgr [165729.761108] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.761110] [ 3951] 0 3951 6464 154 19 58 0 atd [165729.761111] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.761112] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.761140] [67360] 89 67360 25474 215 47 264 0 pickup [165729.761142] [67627] 0 67627 40829 122 33 218 0 crond [165729.761143] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.761145] [67641] 0 67641 79 0 4 19 0 systemd-cgroups [165729.761147] [67646] 0 67646 1 0 1 1 0 systemd-cgroups [165729.761148] Out of memory: Kill process 2429 (sssd) score 0 or sacrifice child [165729.761150] Killed process 2830 (sssd_ssh) total-vm:238452kB, anon-rss:0kB, file-rss:1076kB, shmem-rss:0kB [165729.763550] munged invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [165729.763552] munged cpuset=/ mems_allowed=0-1 [165729.763553] CPU: 5 PID: 3704 Comm: munged Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.763554] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.763555] ffff88040a488fd0 00000000ba5eb381 ffff880413b23920 ffffffff816a3e2d [165729.763556] ffff880413b239b0 ffffffff8169f246 ffffffff810e93ac ffff88041475b2f0 [165729.763557] ffff88041475b308 0000000000000206 0000000000000206 ffff880413b239a0 [165729.763558] Call Trace: [165729.763561] [] dump_stack+0x19/0x1b [165729.763562] [] dump_header+0x90/0x229 [165729.763563] [] ? ktime_get_ts64+0x4c/0xf0 [165729.763564] [] ? delayacct_end+0x8f/0xb0 [165729.763566] [] oom_kill_process+0x254/0x3d0 [165729.763568] [] ? oom_unkillable_task+0xcd/0x120 [165729.763569] [] ? find_lock_task_mm+0x56/0xc0 [165729.763571] [] out_of_memory+0x4b6/0x4f0 [165729.763572] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.763573] [] __alloc_pages_nodemask+0x405/0x420 [165729.763575] [] alloc_pages_current+0x98/0x110 [165729.763576] [] __page_cache_alloc+0x97/0xb0 [165729.763578] [] filemap_fault+0x170/0x410 [165729.763587] [] ext4_filemap_fault+0x36/0x50 [ext4] [165729.763589] [] __do_fault+0x52/0xe0 [165729.763590] [] do_read_fault.isra.44+0x4b/0x130 [165729.763591] [] handle_mm_fault+0x691/0xfa0 [165729.763593] [] __do_page_fault+0x154/0x450 [165729.763594] [] do_page_fault+0x35/0x90 [165729.763596] [] page_fault+0x28/0x30 [165729.763596] Mem-Info: [165729.763601] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:413 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:856 bounce:0 free:39349 free_pcp:783 free_cma:0 [165729.763605] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.763606] lowmem_reserve[]: 0 2580 15620 15620 [165729.763610] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7244 all_unreclaimable? yes [165729.763611] lowmem_reserve[]: 0 0 13040 13040 [165729.763615] Node 0 Normal free:36676kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1400kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:748kB unstable:0kB bounce:0kB free_pcp:1704kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:23334 all_unreclaimable? yes [165729.763615] lowmem_reserve[]: 0 0 0 0 [165729.763619] Node 1 Normal free:45704kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:2612kB unstable:0kB bounce:0kB free_pcp:1324kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29827 all_unreclaimable? yes [165729.763620] lowmem_reserve[]: 0 0 0 0 [165729.763625] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.763630] Node 0 DMA32: 33*4kB (UEM) 1867*8kB (UEM) 1333*16kB (UEM) 504*32kB (UEM) 102*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59436kB [165729.763634] Node 0 Normal: 44*4kB (E) 107*8kB (EM) 111*16kB (UEM) 207*32kB (UEM) 385*64kB (UEM) 13*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 35736kB [165729.763638] Node 1 Normal: 448*4kB (UEM) 391*8kB (UEM) 214*16kB (UEM) 347*32kB (UEM) 349*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45240kB [165729.763639] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.763640] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.763640] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.763641] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.763641] 1785 total pagecache pages [165729.763642] 3 pages in swap cache [165729.763642] Swap cache stats: add 314418, delete 314415, find 93658/151217 [165729.763643] Free swap = 16267704kB [165729.763643] Total swap = 16319484kB [165729.763644] 8369063 pages RAM [165729.763644] 0 pages HighMem/MovableOnly [165729.763644] 241261 pages reserved [165729.763645] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.763651] [ 977] 0 977 20596 225 41 97 0 systemd-journal [165729.763653] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.763654] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.763655] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.763663] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.763664] [ 2429] 0 2429 64920 323 78 279 0 sssd [165729.763665] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.763666] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.763667] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.763668] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.763669] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.763670] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.763672] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.763673] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.763674] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.763675] [ 3875] 0 3875 22907 174 44 280 0 master [165729.763676] [ 3908] 89 3908 25491 216 47 278 0 qmgr [165729.763677] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.763678] [ 3951] 0 3951 6464 154 19 58 0 atd [165729.763679] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.763680] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.763705] [67360] 89 67360 25474 215 47 264 0 pickup [165729.763706] [67627] 0 67627 40829 122 33 218 0 crond [165729.763707] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.763709] [67641] 0 67641 79 0 4 19 0 systemd-cgroups [165729.763710] [67646] 0 67646 1 0 1 1 0 systemd-cgroups [165729.763711] Out of memory: Kill process 2429 (sssd) score 0 or sacrifice child [165729.763712] Killed process 2429 (sssd) total-vm:259680kB, anon-rss:0kB, file-rss:1292kB, shmem-rss:0kB [165729.764782] chronyd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [165729.764783] chronyd cpuset=/ mems_allowed=0-1 [165729.764784] CPU: 28 PID: 2771 Comm: chronyd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.764785] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.764787] ffff88042bb90000 00000000ec76217d ffff88042924b920 ffffffff816a3e2d [165729.764788] ffff88042924b9b0 ffffffff8169f246 ffffffff810e93ac ffff88042d57da60 [165729.764789] ffff88042d57da78 0000000000000206 0000000000000206 ffff88042924b9a0 [165729.764789] Call Trace: [165729.764793] [] dump_stack+0x19/0x1b [165729.764795] [] dump_header+0x90/0x229 [165729.764797] [] ? ktime_get_ts64+0x4c/0xf0 [165729.764798] [] ? delayacct_end+0x8f/0xb0 [165729.764801] [] oom_kill_process+0x254/0x3d0 [165729.764803] [] ? oom_unkillable_task+0xcd/0x120 [165729.764805] [] ? find_lock_task_mm+0x56/0xc0 [165729.764807] [] out_of_memory+0x4b6/0x4f0 [165729.764809] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.764811] [] __alloc_pages_nodemask+0x405/0x420 [165729.764814] [] alloc_pages_current+0x98/0x110 [165729.764816] [] __page_cache_alloc+0x97/0xb0 [165729.764818] [] filemap_fault+0x170/0x410 [165729.764840] [] ext4_filemap_fault+0x36/0x50 [ext4] [165729.764842] [] __do_fault+0x52/0xe0 [165729.764843] [] do_read_fault.isra.44+0x4b/0x130 [165729.764845] [] handle_mm_fault+0x691/0xfa0 [165729.764847] [] ? change_protection+0x65/0xa0 [165729.764849] [] __do_page_fault+0x154/0x450 [165729.764851] [] do_page_fault+0x35/0x90 [165729.764853] [] page_fault+0x28/0x30 [165729.764853] Mem-Info: [165729.764858] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:413 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:856 bounce:0 free:39617 free_pcp:55 free_cma:0 [165729.764862] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.764864] lowmem_reserve[]: 0 2580 15620 15620 [165729.764870] Node 0 DMA32 free:59496kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no [165729.764872] lowmem_reserve[]: 0 0 13040 13040 [165729.764878] Node 0 Normal free:36968kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1400kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:748kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165729.764880] lowmem_reserve[]: 0 0 0 0 [165729.764886] Node 1 Normal free:46424kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:2612kB unstable:0kB bounce:0kB free_pcp:192kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165729.764887] lowmem_reserve[]: 0 0 0 0 [165729.764893] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.764898] Node 0 DMA32: 64*4kB (UEM) 1866*8kB (UEM) 1334*16kB (UEM) 504*32kB (UEM) 102*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59568kB [165729.764905] Node 0 Normal: 189*4kB (UEM) 165*8kB (UEM) 123*16kB (UEM) 219*32kB (UEM) 385*64kB (UEM) 12*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37484kB [165729.764912] Node 1 Normal: 639*4kB (UEM) 403*8kB (UEM) 230*16kB (UEM) 352*32kB (UEM) 350*64kB (UEM) 27*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46580kB [165729.764914] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.764915] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.764916] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.764917] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.764918] 1785 total pagecache pages [165729.764918] 2 pages in swap cache [165729.764919] Swap cache stats: add 314421, delete 314419, find 93659/151219 [165729.764920] Free swap = 16268772kB [165729.764921] Total swap = 16319484kB [165729.764921] 8369063 pages RAM [165729.764922] 0 pages HighMem/MovableOnly [165729.764922] 241261 pages reserved [165729.764923] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.764931] [ 977] 0 977 20596 225 41 97 0 systemd-journal [165729.764933] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.764935] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.764937] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.764945] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.764948] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.764949] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.764951] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.764953] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.764954] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.764956] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.764960] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.764961] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.764963] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.764965] [ 3875] 0 3875 22907 174 44 280 0 master [165729.764966] [ 3908] 89 3908 25491 216 47 278 0 qmgr [165729.764968] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.764969] [ 3951] 0 3951 6464 154 19 58 0 atd [165729.764971] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.764972] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.765000] [67360] 89 67360 25474 215 47 264 0 pickup [165729.765006] [67627] 0 67627 40829 122 33 218 0 crond [165729.765008] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.765010] [67641] 0 67641 79 0 4 19 0 systemd-cgroups [165729.765012] [67646] 0 67646 41 0 2 1 0 systemd-cgroups [165729.765013] Out of memory: Kill process 3908 (qmgr) score 0 or sacrifice child [165729.765015] Killed process 3908 (qmgr) total-vm:101964kB, anon-rss:0kB, file-rss:864kB, shmem-rss:0kB [165729.774314] crond invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [165729.774315] crond cpuset=/ mems_allowed=0-1 [165729.774318] CPU: 24 PID: 3915 Comm: crond Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.774319] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.774321] ffff88082668af70 000000006b346841 ffff88041566f920 ffffffff816a3e2d [165729.774323] ffff88041566f9b0 ffffffff8169f246 ffffffff810e93ac ffff880402e5cf30 [165729.774324] ffff880402e5cf48 0000000000000206 0000000000000206 ffff88041566f9a0 [165729.774325] Call Trace: [165729.774328] [] dump_stack+0x19/0x1b [165729.774330] [] dump_header+0x90/0x229 [165729.774332] [] ? ktime_get_ts64+0x4c/0xf0 [165729.774334] [] ? delayacct_end+0x8f/0xb0 [165729.774336] [] oom_kill_process+0x254/0x3d0 [165729.774338] [] ? oom_unkillable_task+0xcd/0x120 [165729.774340] [] ? find_lock_task_mm+0x56/0xc0 [165729.774343] [] out_of_memory+0x4b6/0x4f0 [165729.774345] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.774347] [] __alloc_pages_nodemask+0x405/0x420 [165729.774349] [] alloc_pages_current+0x98/0x110 [165729.774351] [] __page_cache_alloc+0x97/0xb0 [165729.774353] [] filemap_fault+0x170/0x410 [165729.774366] [] ext4_filemap_fault+0x36/0x50 [ext4] [165729.774367] [] __do_fault+0x52/0xe0 [165729.774369] [] do_read_fault.isra.44+0x4b/0x130 [165729.774371] [] handle_mm_fault+0x691/0xfa0 [165729.774373] [] __do_page_fault+0x154/0x450 [165729.774375] [] do_page_fault+0x35/0x90 [165729.774377] [] page_fault+0x28/0x30 [165729.774378] Mem-Info: [165729.774385] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:192 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:856 bounce:0 free:39421 free_pcp:0 free_cma:0 [165729.774391] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.774393] lowmem_reserve[]: 0 2580 15620 15620 [165729.774399] Node 0 DMA32 free:59476kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:141 all_unreclaimable? yes [165729.774401] lowmem_reserve[]: 0 0 13040 13040 [165729.774407] Node 0 Normal free:36968kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:516kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:748kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1336 all_unreclaimable? yes [165729.774408] lowmem_reserve[]: 0 0 0 0 [165729.774415] Node 1 Normal free:45660kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:2612kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:432 all_unreclaimable? yes [165729.774416] lowmem_reserve[]: 0 0 0 0 [165729.774425] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.774433] Node 0 DMA32: 63*4kB (UEM) 1864*8kB (UEM) 1333*16kB (UEM) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165729.774440] Node 0 Normal: 185*4kB (UEM) 165*8kB (UEM) 120*16kB (UEM) 222*32kB (UEM) 385*64kB (UEM) 12*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37516kB [165729.774447] Node 1 Normal: 685*4kB (UEM) 405*8kB (UEM) 230*16kB (UEM) 352*32kB (UEM) 350*64kB (UEM) 22*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46140kB [165729.774449] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.774450] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.774451] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.774452] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.774453] 1785 total pagecache pages [165729.774453] 5 pages in swap cache [165729.774454] Swap cache stats: add 314440, delete 314435, find 93663/151227 [165729.774455] Free swap = 16269796kB [165729.774456] Total swap = 16319484kB [165729.774457] 8369063 pages RAM [165729.774457] 0 pages HighMem/MovableOnly [165729.774457] 241261 pages reserved [165729.774458] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.774466] [ 977] 0 977 20596 225 41 98 0 systemd-journal [165729.774468] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.774470] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.774471] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.774480] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.774482] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.774484] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.774486] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.774488] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.774489] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.774491] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.774494] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.774495] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.774497] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.774499] [ 3875] 0 3875 22907 174 44 281 0 master [165729.774500] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.774502] [ 3951] 0 3951 6464 154 19 59 0 atd [165729.774504] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.774505] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.774532] [67360] 89 67360 25474 215 47 264 0 pickup [165729.774534] [67627] 0 67627 40829 122 33 219 0 crond [165729.774536] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.774538] [67641] 0 67641 79 0 4 20 0 systemd-cgroups [165729.774540] [67646] 0 67646 76 0 3 2 0 systemd-cgroups [165729.774541] Out of memory: Kill process 67360 (pickup) score 0 or sacrifice child [165729.774543] Killed process 67360 (pickup) total-vm:101896kB, anon-rss:0kB, file-rss:860kB, shmem-rss:0kB [165729.779338] munged invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165729.779339] munged cpuset=/ mems_allowed=0-1 [165729.779341] CPU: 6 PID: 3704 Comm: munged Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.779342] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.779343] ffff88040a488fd0 00000000ba5eb381 ffff880413b23988 ffffffff816a3e2d [165729.779344] ffff880413b23a18 ffffffff8169f246 ffffffff810e93ac ffff88041475b2f0 [165729.779346] ffff88041475b308 0000000000000202 0000000000000202 ffff880413b23a08 [165729.779346] Call Trace: [165729.779348] [] dump_stack+0x19/0x1b [165729.779349] [] dump_header+0x90/0x229 [165729.779351] [] ? ktime_get_ts64+0x4c/0xf0 [165729.779352] [] ? delayacct_end+0x8f/0xb0 [165729.779354] [] oom_kill_process+0x254/0x3d0 [165729.779356] [] ? oom_unkillable_task+0xcd/0x120 [165729.779357] [] ? find_lock_task_mm+0x56/0xc0 [165729.779359] [] out_of_memory+0x4b6/0x4f0 [165729.779360] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.779362] [] __alloc_pages_nodemask+0x405/0x420 [165729.779363] [] alloc_pages_vma+0xb5/0x200 [165729.779365] [] read_swap_cache_async+0xed/0x160 [165729.779366] [] ? mem_cgroup_update_page_stat+0x16/0x50 [165729.779367] [] swapin_readahead+0xa8/0x110 [165729.779368] [] handle_mm_fault+0xadb/0xfa0 [165729.779370] [] __do_page_fault+0x154/0x450 [165729.779372] [] do_page_fault+0x35/0x90 [165729.779373] [] page_fault+0x28/0x30 [165729.779374] Mem-Info: [165729.779377] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:192 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:856 bounce:0 free:39421 free_pcp:113 free_cma:0 [165729.779381] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.779383] lowmem_reserve[]: 0 2580 15620 15620 [165729.779387] Node 0 DMA32 free:59476kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:177 all_unreclaimable? yes [165729.779388] lowmem_reserve[]: 0 0 13040 13040 [165729.779391] Node 0 Normal free:36968kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:516kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:748kB unstable:0kB bounce:0kB free_pcp:236kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1348 all_unreclaimable? yes [165729.779392] lowmem_reserve[]: 0 0 0 0 [165729.779396] Node 1 Normal free:45660kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:2612kB unstable:0kB bounce:0kB free_pcp:216kB local_pcp:4kB free_cma:0kB writeback_tmp:0kB pages_scanned:452 all_unreclaimable? yes [165729.779397] lowmem_reserve[]: 0 0 0 0 [165729.779402] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.779406] Node 0 DMA32: 63*4kB (UEM) 1864*8kB (UEM) 1333*16kB (UEM) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165729.779411] Node 0 Normal: 123*4kB (UEM) 165*8kB (UEM) 120*16kB (UEM) 222*32kB (UEM) 385*64kB (UEM) 12*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37268kB [165729.779415] Node 1 Normal: 685*4kB (UEM) 405*8kB (UEM) 230*16kB (UEM) 352*32kB (UEM) 350*64kB (UEM) 22*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46140kB [165729.779416] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.779417] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.779417] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.779418] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.779418] 1785 total pagecache pages [165729.779419] 7 pages in swap cache [165729.779419] Swap cache stats: add 314443, delete 314436, find 93665/151231 [165729.779420] Free swap = 16270808kB [165729.779420] Total swap = 16319484kB [165729.779421] 8369063 pages RAM [165729.779421] 0 pages HighMem/MovableOnly [165729.779421] 241261 pages reserved [165729.779422] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.779428] [ 977] 0 977 20596 225 41 98 0 systemd-journal [165729.779429] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.779430] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.779432] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.779439] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.779440] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.779441] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.779442] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.779443] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.779444] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.779445] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.779447] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.779448] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.779449] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.779450] [ 3875] 0 3875 22907 174 44 281 0 master [165729.779451] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.779452] [ 3951] 0 3951 6464 154 19 59 0 atd [165729.779453] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.779454] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.779479] [67627] 0 67627 40829 122 33 219 0 crond [165729.779481] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.779482] [67641] 0 67641 79 0 4 20 0 systemd-cgroups [165729.779483] [67646] 0 67646 76 0 3 2 0 systemd-cgroups [165729.779484] Out of memory: Kill process 3875 (master) score 0 or sacrifice child [165729.779485] Killed process 3875 (master) total-vm:91628kB, anon-rss:0kB, file-rss:696kB, shmem-rss:0kB [165729.788457] kworker/u482:2 invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 [165729.788458] kworker/u482:2 cpuset=/ mems_allowed=0-1 [165729.788461] CPU: 11 PID: 67640 Comm: kworker/u482:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.788461] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.788465] Workqueue: events_unbound __call_usermodehelper [165729.788467] ffff88081bc39fa0 00000000137ddf5c ffff88042b0d79a0 ffffffff816a3e2d [165729.788469] ffff88042b0d7a30 ffffffff8169f246 ffffffff810e93ac ffff8807ebebe280 [165729.788470] ffff8807ebebe298 0000000000000206 0000000000000206 ffff88042b0d7a20 [165729.788471] Call Trace: [165729.788474] [] dump_stack+0x19/0x1b [165729.788475] [] dump_header+0x90/0x229 [165729.788477] [] ? ktime_get_ts64+0x4c/0xf0 [165729.788479] [] ? delayacct_end+0x8f/0xb0 [165729.788481] [] oom_kill_process+0x254/0x3d0 [165729.788483] [] ? oom_unkillable_task+0xcd/0x120 [165729.788484] [] ? find_lock_task_mm+0x56/0xc0 [165729.788486] [] out_of_memory+0x4b6/0x4f0 [165729.788488] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.788490] [] __alloc_pages_nodemask+0x405/0x420 [165729.788492] [] ? ____call_usermodehelper+0x130/0x130 [165729.788496] [] copy_process+0x1dd/0x1970 [165729.788499] [] ? account_entity_dequeue+0xae/0xd0 [165729.788501] [] do_fork+0x91/0x320 [165729.788503] [] ? __switch_to+0xd7/0x510 [165729.788505] [] kernel_thread+0x26/0x30 [165729.788507] [] __call_usermodehelper+0x2e/0x90 [165729.788509] [] process_one_work+0x17a/0x440 [165729.788511] [] worker_thread+0x126/0x3c0 [165729.788513] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165729.788514] [] kthread+0xcf/0xe0 [165729.788516] [] ? insert_kthread_work+0x40/0x40 [165729.788518] [] ret_from_fork+0x58/0x90 [165729.788519] [] ? insert_kthread_work+0x40/0x40 [165729.788520] Mem-Info: [165729.788525] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:404 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:856 bounce:0 free:39402 free_pcp:0 free_cma:0 [165729.788530] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165729.788532] lowmem_reserve[]: 0 2580 15620 15620 [165729.788537] Node 0 DMA32 free:59476kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1632kB pagetables:64kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:222 all_unreclaimable? yes [165729.788539] lowmem_reserve[]: 0 0 13040 13040 [165729.788544] Node 0 Normal free:36968kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1364kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19408kB slab_unreclaimable:1089644kB kernel_stack:14176kB pagetables:748kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2122 all_unreclaimable? yes [165729.788545] lowmem_reserve[]: 0 0 0 0 [165729.788550] Node 1 Normal free:45584kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:2612kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:372 all_unreclaimable? yes [165729.788551] lowmem_reserve[]: 0 0 0 0 [165729.788558] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165729.788565] Node 0 DMA32: 63*4kB (UEM) 1864*8kB (UEM) 1333*16kB (UEM) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59564kB [165729.788571] Node 0 Normal: 215*4kB (UEM) 172*8kB (UEM) 120*16kB (UEM) 222*32kB (UEM) 385*64kB (UEM) 12*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37692kB [165729.788577] Node 1 Normal: 746*4kB (UEM) 405*8kB (UEM) 229*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 20*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46144kB [165729.788578] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.788579] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.788580] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.788581] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.788581] 1785 total pagecache pages [165729.788582] 4 pages in swap cache [165729.788583] Swap cache stats: add 314444, delete 314440, find 93666/151234 [165729.788583] Free swap = 16271868kB [165729.788583] Total swap = 16319484kB [165729.788584] 8369063 pages RAM [165729.788584] 0 pages HighMem/MovableOnly [165729.788585] 241261 pages reserved [165729.788585] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.788592] [ 977] 0 977 20596 225 41 98 0 systemd-journal [165729.788594] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.788595] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.788597] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.788605] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.788607] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.788608] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.788609] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.788611] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.788613] [ 2518] 0 2518 32859 203 31 231 0 rpc.gssd [165729.788614] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.788617] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.788619] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.788620] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.788622] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.788623] [ 3951] 0 3951 6464 154 19 59 0 atd [165729.788625] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.788626] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.788654] [67627] 0 67627 40829 122 33 220 0 crond [165729.788655] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.788657] [67641] 0 67641 79 0 4 20 0 systemd-cgroups [165729.788659] [67646] 0 67646 76 0 3 2 0 systemd-cgroups [165729.788660] Out of memory: Kill process 2518 (rpc.gssd) score 0 or sacrifice child [165729.788661] Killed process 2518 (rpc.gssd) total-vm:131436kB, anon-rss:0kB, file-rss:812kB, shmem-rss:0kB [165729.798677] ll_ost_out01_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [165729.798678] ll_ost_out01_00 cpuset=/ mems_allowed=1 [165729.798680] CPU: 8 PID: 183717 Comm: ll_ost_out01_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165729.798681] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165729.798683] ffff8803dd56eeb0 0000000085bdad45 ffff8803966af998 ffffffff816a3e2d [165729.798685] ffff8803966afa28 ffffffff8169f246 ffffffff810e93ac ffff8807f1e228a0 [165729.798686] ffff8807f1e228b8 0000000000000206 0000000000000206 ffff8803966afa18 [165729.798687] Call Trace: [165729.798690] [] dump_stack+0x19/0x1b [165729.798691] [] dump_header+0x90/0x229 [165729.798693] [] ? ktime_get_ts64+0x4c/0xf0 [165729.798694] [] ? delayacct_end+0x8f/0xb0 [165729.798696] [] oom_kill_process+0x254/0x3d0 [165729.798698] [] ? oom_unkillable_task+0xcd/0x120 [165729.798700] [] ? find_lock_task_mm+0x56/0xc0 [165729.798702] [] out_of_memory+0x4b6/0x4f0 [165729.798704] [] __alloc_pages_slowpath+0x5d6/0x724 [165729.798705] [] __alloc_pages_nodemask+0x405/0x420 [165729.798707] [] __vmalloc_node_range+0x12f/0x270 [165729.798750] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165729.798752] [] vzalloc_node+0x47/0x50 [165729.798782] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165729.798812] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165729.798841] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [165729.798872] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [165729.798901] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [165729.798903] [] kthread+0xcf/0xe0 [165729.798905] [] ? insert_kthread_work+0x40/0x40 [165729.798906] [] ret_from_fork+0x58/0x90 [165729.798907] [] ? insert_kthread_work+0x40/0x40 [165729.798908] Mem-Info: [165729.798912] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:298 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12097 slab_unreclaimable:560306 mapped:1547 shmem:0 pagetables:856 bounce:0 free:39305 free_pcp:0 free_cma:0 [165729.798918] Node 1 Normal free:45196kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:4kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25880kB slab_unreclaimable:957392kB kernel_stack:16624kB pagetables:2612kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:594 all_unreclaimable? yes [165729.798920] lowmem_reserve[]: 0 0 0 0 [165729.798926] Node 1 Normal: 773*4kB (UEM) 406*8kB (UEM) 229*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 20*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46260kB [165729.798927] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.798928] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.798929] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165729.798929] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165729.798930] 1785 total pagecache pages [165729.798930] 4 pages in swap cache [165729.798931] Swap cache stats: add 314452, delete 314448, find 93667/151236 [165729.798932] Free swap = 16272792kB [165729.798932] Total swap = 16319484kB [165729.798933] 8369063 pages RAM [165729.798934] 0 pages HighMem/MovableOnly [165729.798934] 241261 pages reserved [165729.798935] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165729.798942] [ 977] 0 977 20596 225 41 98 0 systemd-journal [165729.798944] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165729.798945] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165729.798947] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165729.798955] [ 2373] 0 2373 13863 116 27 117 -1000 auditd [165729.798957] [ 2434] 0 2434 67755 205 45 178 0 gssproxy [165729.798958] [ 2435] 0 2435 5461 182 15 150 0 irqbalance [165729.798960] [ 2444] 81 2444 9196 203 21 147 -900 dbus-daemon [165729.798962] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165729.798964] [ 2771] 998 2771 29553 242 29 126 0 chronyd [165729.798966] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165729.798968] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165729.798969] [ 3703] 996 3703 24211 164 21 136 0 munged [165729.798971] [ 3915] 0 3915 31559 191 18 172 0 crond [165729.798972] [ 3951] 0 3951 6464 154 19 59 0 atd [165729.798974] [ 3970] 0 3970 27511 156 12 32 0 agetty [165729.798975] [ 3971] 0 3971 27511 160 10 31 0 agetty [165729.799005] [67627] 0 67627 40829 122 33 220 0 crond [165729.799007] [67635] 0 67635 90 16 4 11 0 systemd-cgroups [165729.799009] [67641] 0 67641 79 0 4 20 0 systemd-cgroups [165729.799011] [67646] 0 67646 76 0 3 2 0 systemd-cgroups [165729.799012] Out of memory: Kill process 2434 (gssproxy) score 0 or sacrifice child [165729.799014] Killed process 2434 (gssproxy) total-vm:271020kB, anon-rss:0kB, file-rss:820kB, shmem-rss:0kB [165729.822860] Out of memory: Kill process 2771 (chronyd) score 0 or sacrifice child [165729.822861] Killed process 2771 (chronyd) total-vm:118212kB, anon-rss:0kB, file-rss:964kB, shmem-rss:0kB [165755.601933] oom_kill_process: 1 callbacks suppressed [165755.619411] irqbalance invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165755.646321] irqbalance cpuset=/ mems_allowed=0-1 [165755.662737] CPU: 3 PID: 2435 Comm: irqbalance Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165755.700040] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165755.738089] ffff880417819fa0 0000000011252cbf ffff880813ebf988 ffffffff816a3e2d [165755.763705] ffff880813ebfa18 ffffffff8169f246 000000000000000a 0000000000000000 [165755.789180] 0000000000000001 0000000000000206 0000000000000206 ffffffffffffff10 [165755.814700] Call Trace: [165755.823876] [] dump_stack+0x19/0x1b [165755.841928] [] dump_header+0x90/0x229 [165755.860586] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [165755.884142] [] oom_kill_process+0x254/0x3d0 [165755.904365] [] ? oom_unkillable_task+0xcd/0x120 [165755.925856] [] ? find_lock_task_mm+0x56/0xc0 [165755.946437] [] out_of_memory+0x4b6/0x4f0 [165755.965953] [] __alloc_pages_slowpath+0x5d6/0x724 [165755.987960] [] __alloc_pages_nodemask+0x405/0x420 [165756.010054] [] alloc_pages_vma+0xb5/0x200 [165756.029838] [] read_swap_cache_async+0xed/0x160 [165756.051216] [] ? mem_cgroup_update_page_stat+0x16/0x50 [165756.074759] [] swapin_readahead+0xa8/0x110 [165756.094861] [] handle_mm_fault+0xadb/0xfa0 [165756.114938] [] __do_page_fault+0x154/0x450 [165756.134964] [] do_page_fault+0x35/0x90 [165756.153875] [] page_fault+0x28/0x30 [165756.171947] Mem-Info: [165756.180584] active_anon:0 inactive_anon:5 isolated_anon:0 active_file:31 inactive_file:190 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12091 slab_unreclaimable:560268 mapped:1547 shmem:0 pagetables:543 bounce:0 free:39232 free_pcp:0 free_cma:0 [165756.293267] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165756.433847] lowmem_reserve[]: 0 2580 15620 15620 [165756.450463] Node 0 DMA32 free:59464kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194188kB kernel_stack:1584kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5399 all_unreclaimable? no [165756.601697] lowmem_reserve[]: 0 0 13040 13040 [165756.617627] Node 0 Normal free:36780kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:504kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089644kB kernel_stack:14112kB pagetables:412kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1304 all_unreclaimable? yes [165756.774329] lowmem_reserve[]: 0 0 0 0 [165756.788007] Node 1 Normal free:45264kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:196kB inactive_file:408kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957240kB kernel_stack:16624kB pagetables:1736kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1170 all_unreclaimable? yes [165756.946068] lowmem_reserve[]: 0 0 0 0 [165756.959789] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165757.007195] Node 0 DMA32: 50*4kB (UEM) 1860*8kB (UEM) 1334*16kB (UEM) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165757.059297] Node 0 Normal: 127*4kB (UEM) 169*8kB (UEM) 126*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36900kB [165757.113236] Node 1 Normal: 595*4kB (UEM) 405*8kB (UEM) 236*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45268kB [165757.165900] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165757.196414] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165757.226056] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165757.256544] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165757.286160] 1782 total pagecache pages [165757.300335] 2 pages in swap cache [165757.313080] Swap cache stats: add 314573, delete 314571, find 93708/151308 [165757.337563] Free swap = 16273884kB [165757.350887] Total swap = 16319484kB [165757.364215] 8369063 pages RAM [165757.375819] 0 pages HighMem/MovableOnly [165757.390270] 241261 pages reserved [165757.403006] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165757.430370] [ 977] 0 977 20596 224 41 102 0 systemd-journal [165757.461169] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165757.489576] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165757.519648] [ 1046] 0 1046 1588718 3697 132 0 -1000 multipathd [165757.548802] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165757.576753] [ 2435] 0 2435 5461 181 15 151 0 irqbalance [165757.606045] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165757.635514] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165757.663385] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165757.691450] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165757.718934] [ 3703] 996 3703 24211 162 21 138 0 munged [165757.746751] [ 3915] 0 3915 31559 189 18 173 0 crond [165757.774132] [ 3951] 0 3951 6464 152 19 61 0 atd [165757.800967] [ 3970] 0 3970 27511 156 12 32 0 agetty [165757.828707] [ 3971] 0 3971 27511 160 10 31 0 agetty [165757.856641] [67627] 0 67627 40829 120 33 222 0 crond [165757.884199] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165757.914523] [67641] 0 67641 79 0 4 27 0 systemd-cgroups [165757.944657] [67646] 0 67646 79 0 4 5 0 systemd-cgroups [165757.974738] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165758.004807] [67648] 0 67648 79 0 3 3 0 systemd-cgroups [165758.035095] [67649] 0 67649 79 0 3 3 0 systemd-cgroups [165758.065119] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165758.095037] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165758.124918] [67652] 0 67652 76 0 3 2 0 systemd-cgroups [165758.154841] Out of memory: Kill process 3915 (crond) score 0 or sacrifice child [165758.180112] Killed process 67627 (crond) total-vm:163316kB, anon-rss:0kB, file-rss:480kB, shmem-rss:0kB [165758.213486] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165758.242239] systemd-cgroups cpuset=/ mems_allowed=0-1 [165758.260089] CPU: 31 PID: 67651 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165758.300257] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165758.338380] ffff8806a57deeb0 000000003cffd884 ffff88042b12b658 ffffffff816a3e2d [165758.363790] ffff88042b12b6e8 ffffffff8169f246 ffffffff810e93ac ffff880036a28980 [165758.389125] ffff880036a28998 0000000000000206 0000000000000206 ffff88042b12b6d8 [165758.414599] Call Trace: [165758.423726] [] dump_stack+0x19/0x1b [165758.441764] [] dump_header+0x90/0x229 [165758.460340] [] ? ktime_get_ts64+0x4c/0xf0 [165758.480072] [] ? delayacct_end+0x8f/0xb0 [165758.499505] [] oom_kill_process+0x254/0x3d0 [165758.519783] [] ? oom_unkillable_task+0xcd/0x120 [165758.541234] [] ? find_lock_task_mm+0x56/0xc0 [165758.561792] [] out_of_memory+0x4b6/0x4f0 [165758.581216] [] __alloc_pages_slowpath+0x5d6/0x724 [165758.603208] [] __alloc_pages_nodemask+0x405/0x420 [165758.625235] [] alloc_pages_vma+0xb5/0x200 [165758.644974] [] read_swap_cache_async+0xed/0x160 [165758.666398] [] swapin_readahead+0xa8/0x110 [165758.686348] [] handle_mm_fault+0xadb/0xfa0 [165758.706307] [] ? mmap_region+0x1ba/0x640 [165758.725718] [] __do_page_fault+0x154/0x450 [165758.745700] [] do_page_fault+0x35/0x90 [165758.764528] [] page_fault+0x28/0x30 [165758.782511] [] ? strnlen_user+0x3e/0x100 [165758.801937] [] ? create_elf_tables+0x407/0x523 [165758.823087] [] load_elf_binary+0xa00/0xe00 [165758.843021] [] ? load_elf_library+0x220/0x220 [165758.863824] [] search_binary_handler+0xed/0x300 [165758.885238] [] do_execve_common.isra.25+0x5b6/0x6c0 [165758.907786] [] do_execve+0x18/0x20 [165758.925490] [] ____call_usermodehelper+0xfc/0x130 [165758.947438] [] ? ____call_usermodehelper+0x130/0x130 [165758.970293] [] call_helper+0x1e/0x20 [165758.988562] [] ret_from_fork+0x58/0x90 [165759.007441] [] ? ____call_usermodehelper+0x130/0x130 [165759.030325] Mem-Info: [165759.038901] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:99 inactive_file:135 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12083 slab_unreclaimable:560256 mapped:1546 shmem:0 pagetables:512 bounce:0 free:39324 free_pcp:0 free_cma:0 [165759.151759] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165759.291923] lowmem_reserve[]: 0 2580 15620 15620 [165759.308336] Node 0 DMA32 free:58992kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:8kB active_file:60kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:336kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194180kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:488kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3227 all_unreclaimable? yes [165759.459034] lowmem_reserve[]: 0 0 13040 13040 [165759.474681] Node 0 Normal free:36476kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:116kB active_file:180kB inactive_file:248kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:88kB mapped:1536kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089612kB kernel_stack:14112kB pagetables:376kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2174 all_unreclaimable? yes [165759.631535] lowmem_reserve[]: 0 0 0 0 [165759.645004] Node 1 Normal free:45336kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957232kB kernel_stack:16624kB pagetables:1676kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1488 all_unreclaimable? yes [165759.802374] lowmem_reserve[]: 0 0 0 0 [165759.816096] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165759.863504] Node 0 DMA32: 47*4kB (UEM) 1859*8kB (UEM) 1333*16kB (UEM) 502*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59492kB [165759.915104] Node 0 Normal: 127*4kB (EM) 158*8kB (UEM) 118*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36684kB [165759.968561] Node 1 Normal: 608*4kB (UEM) 407*8kB (UEM) 236*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45336kB [165760.020721] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165760.051142] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165760.080769] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165760.111076] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165760.140529] 1843 total pagecache pages [165760.154633] 14 pages in swap cache [165760.167486] Swap cache stats: add 314657, delete 314643, find 93724/151338 [165760.191923] Free swap = 16274088kB [165760.204910] Total swap = 16319484kB [165760.218021] 8369063 pages RAM [165760.229362] 0 pages HighMem/MovableOnly [165760.243706] 241261 pages reserved [165760.256262] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165760.283783] [ 977] 0 977 20596 223 41 103 0 systemd-journal [165760.314483] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165760.342897] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165760.373019] [ 1046] 0 1046 1588718 3687 132 0 -1000 multipathd [165760.402253] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165760.430694] [ 2435] 0 2435 5461 181 15 151 0 irqbalance [165760.459592] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165760.488787] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165760.516537] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165760.544269] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165760.571411] [ 3703] 996 3703 24211 161 21 139 0 munged [165760.599075] [ 3915] 0 3915 31559 188 18 174 0 crond [165760.626456] [ 3951] 0 3951 6464 151 19 62 0 atd [165760.653233] [ 3970] 0 3970 27511 156 12 32 0 agetty [165760.680842] [ 3971] 0 3971 27511 160 10 31 0 agetty [165760.708484] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165760.738634] [67641] 0 67641 79 0 4 28 0 systemd-cgroups [165760.768751] [67646] 0 67646 610 26 4 10 0 systemd-cgroups [165760.798834] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165760.828891] [67648] 0 67648 610 26 4 13 0 systemd-cgroups [165760.858939] [67649] 0 67649 79 0 3 5 0 systemd-cgroups [165760.888963] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165760.918955] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165760.948919] [67652] 0 67652 610 29 5 6 0 systemd-cgroups [165760.978854] [67654] 0 67654 610 26 4 10 0 systemd-cgroups [165761.008761] Out of memory: Kill process 3915 (crond) score 0 or sacrifice child [165761.033812] Killed process 3915 (crond) total-vm:126236kB, anon-rss:0kB, file-rss:752kB, shmem-rss:0kB [165761.079910] systemd-cgroups invoked oom-killer: gfp_mask=0x2084d0, order=0, oom_score_adj=0 [165761.079919] systemd-cgroups cpuset=/ mems_allowed=0-1 [165761.079921] CPU: 30 PID: 67649 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.079922] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.079925] ffff8805acfabf40 00000000ac06895f ffff88042d8a7a48 ffffffff816a3e2d [165761.079926] ffff88042d8a7ad8 ffffffff8169f246 ffffffff810e93ac ffff8802b1253b40 [165761.079928] ffff8802b1253b58 0000000000000206 0000000000000206 ffff88042d8a7ac8 [165761.079929] Call Trace: [165761.079932] [] dump_stack+0x19/0x1b [165761.079933] [] dump_header+0x90/0x229 [165761.079935] [] ? ktime_get_ts64+0x4c/0xf0 [165761.079936] [] ? delayacct_end+0x8f/0xb0 [165761.079938] [] oom_kill_process+0x254/0x3d0 [165761.079940] [] ? oom_unkillable_task+0xcd/0x120 [165761.079941] [] ? find_lock_task_mm+0x56/0xc0 [165761.079943] [] out_of_memory+0x4b6/0x4f0 [165761.079945] [] __alloc_pages_slowpath+0x5d6/0x724 [165761.079946] [] __alloc_pages_nodemask+0x405/0x420 [165761.079948] [] alloc_pages_current+0x98/0x110 [165761.079950] [] pte_alloc_one+0x17/0x40 [165761.079952] [] __pte_alloc+0x23/0x170 [165761.079954] [] handle_mm_fault+0xe17/0xfa0 [165761.079958] [] ? kmem_cache_free+0x1e2/0x200 [165761.079960] [] ? putname+0x3d/0x60 [165761.079962] [] __do_page_fault+0x154/0x450 [165761.079963] [] do_page_fault+0x35/0x90 [165761.079964] [] page_fault+0x28/0x30 [165761.079965] Mem-Info: [165761.079969] active_anon:1 inactive_anon:13 isolated_anon:0 active_file:107 inactive_file:108 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12083 slab_unreclaimable:560256 mapped:1548 shmem:0 pagetables:519 bounce:0 free:39267 free_pcp:0 free_cma:0 [165761.079973] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.079974] lowmem_reserve[]: 0 2580 15620 15620 [165761.079979] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194180kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4352 all_unreclaimable? yes [165761.079981] lowmem_reserve[]: 0 0 13040 13040 [165761.079986] Node 0 Normal free:36684kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:48kB active_file:176kB inactive_file:52kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089612kB kernel_stack:14112kB pagetables:376kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1060 all_unreclaimable? yes [165761.079987] lowmem_reserve[]: 0 0 0 0 [165761.079991] Node 1 Normal free:45336kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957232kB kernel_stack:16624kB pagetables:1676kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1093 all_unreclaimable? yes [165761.079992] lowmem_reserve[]: 0 0 0 0 [165761.079997] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.080002] Node 0 DMA32: 48*4kB (UEM) 1861*8kB (UEM) 1334*16kB (UEM) 503*32kB (UEM) 103*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165761.080008] Node 0 Normal: 147*4kB (UEM) 164*8kB (UEM) 122*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36876kB [165761.080014] Node 1 Normal: 608*4kB (UEM) 407*8kB (UEM) 236*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45336kB [165761.080015] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.080015] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.080016] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.080017] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.080017] 1797 total pagecache pages [165761.080017] 4 pages in swap cache [165761.080018] Swap cache stats: add 314673, delete 314669, find 93725/151341 [165761.080019] Free swap = 16274664kB [165761.080019] Total swap = 16319484kB [165761.080020] 8369063 pages RAM [165761.080020] 0 pages HighMem/MovableOnly [165761.080020] 241261 pages reserved [165761.080020] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165761.080027] [ 977] 0 977 20596 223 41 103 0 systemd-journal [165761.080029] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165761.080030] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165761.080032] [ 1046] 0 1046 1588718 3684 132 0 -1000 multipathd [165761.080040] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165761.080041] [ 2435] 0 2435 5461 181 15 151 0 irqbalance [165761.080042] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165761.080044] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165761.080046] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165761.080047] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165761.080048] [ 3703] 996 3703 24211 161 21 139 0 munged [165761.080050] [ 3951] 0 3951 6464 151 19 62 0 atd [165761.080050] [ 3970] 0 3970 27511 156 12 32 0 agetty [165761.080051] [ 3971] 0 3971 27511 160 10 31 0 agetty [165761.080077] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165761.080079] [67641] 0 67641 79 0 4 28 0 systemd-cgroups [165761.080080] [67646] 0 67646 610 26 4 10 0 systemd-cgroups [165761.080081] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165761.080083] [67648] 0 67648 610 26 4 13 0 systemd-cgroups [165761.080084] [67649] 0 67649 79 0 3 5 0 systemd-cgroups [165761.080086] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165761.080087] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165761.080088] [67652] 0 67652 610 26 5 9 0 systemd-cgroups [165761.080089] [67654] 0 67654 610 26 4 10 0 systemd-cgroups [165761.080090] Out of memory: Kill process 977 (systemd-journal) score 0 or sacrifice child [165761.080091] Killed process 977 (systemd-journal) total-vm:82384kB, anon-rss:0kB, file-rss:892kB, shmem-rss:0kB [165761.191997] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165761.191999] systemd-cgroups cpuset=/ mems_allowed=0-1 [165761.192001] CPU: 23 PID: 67635 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.192001] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.192003] ffff8802c1f08000 000000008d373a7b ffff8802763b3988 ffffffff816a3e2d [165761.192005] ffff8802763b3a18 ffffffff8169f246 ffffffff810e93ac ffff88041475b600 [165761.192006] ffff88041475b618 0000000000000202 0000000000000202 ffff8802763b3a08 [165761.192007] Call Trace: [165761.192010] [] dump_stack+0x19/0x1b [165761.192012] [] dump_header+0x90/0x229 [165761.192014] [] ? ktime_get_ts64+0x4c/0xf0 [165761.192016] [] ? delayacct_end+0x8f/0xb0 [165761.192018] [] oom_kill_process+0x254/0x3d0 [165761.192020] [] ? oom_unkillable_task+0xcd/0x120 [165761.192021] [] ? find_lock_task_mm+0x56/0xc0 [165761.192023] [] out_of_memory+0x4b6/0x4f0 [165761.192025] [] __alloc_pages_slowpath+0x5d6/0x724 [165761.192027] [] __alloc_pages_nodemask+0x405/0x420 [165761.192029] [] alloc_pages_vma+0xb5/0x200 [165761.192030] [] ? end_swap_bio_write+0x80/0x80 [165761.192032] [] read_swap_cache_async+0xed/0x160 [165761.192033] [] swapin_readahead+0xa8/0x110 [165761.192035] [] handle_mm_fault+0xadb/0xfa0 [165761.192038] [] __do_page_fault+0x154/0x450 [165761.192039] [] do_page_fault+0x35/0x90 [165761.192041] [] page_fault+0x28/0x30 [165761.192042] Mem-Info: [165761.192047] active_anon:1 inactive_anon:13 isolated_anon:0 active_file:225 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12083 slab_unreclaimable:560256 mapped:1548 shmem:0 pagetables:519 bounce:0 free:39267 free_pcp:0 free_cma:0 [165761.192052] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.192054] lowmem_reserve[]: 0 2580 15620 15620 [165761.192059] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194180kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4400 all_unreclaimable? yes [165761.192060] lowmem_reserve[]: 0 0 13040 13040 [165761.192066] Node 0 Normal free:36684kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:48kB active_file:648kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089612kB kernel_stack:14112kB pagetables:376kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:45997 all_unreclaimable? yes [165761.192067] lowmem_reserve[]: 0 0 0 0 [165761.192072] Node 1 Normal free:45336kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957232kB kernel_stack:16624kB pagetables:1676kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1309 all_unreclaimable? yes [165761.192073] lowmem_reserve[]: 0 0 0 0 [165761.192082] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.192088] Node 0 DMA32: 48*4kB (UEM) 1861*8kB (UEM) 1334*16kB (UEM) 503*32kB (UEM) 103*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165761.192095] Node 0 Normal: 155*4kB (UEM) 166*8kB (UEM) 122*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36924kB [165761.192102] Node 1 Normal: 636*4kB (UEM) 410*8kB (UEM) 238*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45504kB [165761.192103] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.192104] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.192105] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.192106] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.192106] 1797 total pagecache pages [165761.192107] 1 pages in swap cache [165761.192108] Swap cache stats: add 314679, delete 314678, find 93726/151343 [165761.192108] Free swap = 16275028kB [165761.192109] Total swap = 16319484kB [165761.192110] 8369063 pages RAM [165761.192110] 0 pages HighMem/MovableOnly [165761.192110] 241261 pages reserved [165761.192111] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165761.192119] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165761.192121] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165761.192122] [ 1046] 0 1046 1588718 3684 132 0 -1000 multipathd [165761.192132] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165761.192134] [ 2435] 0 2435 5461 181 15 151 0 irqbalance [165761.192136] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165761.192137] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165761.192141] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165761.192142] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165761.192143] [ 3703] 996 3703 24211 161 21 139 0 munged [165761.192145] [ 3951] 0 3951 6464 151 19 62 0 atd [165761.192147] [ 3970] 0 3970 27511 156 12 32 0 agetty [165761.192148] [ 3971] 0 3971 27511 160 10 31 0 agetty [165761.192176] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165761.192177] [67641] 0 67641 79 0 4 28 0 systemd-cgroups [165761.192179] [67646] 0 67646 610 26 4 10 0 systemd-cgroups [165761.192180] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165761.192182] [67648] 0 67648 610 26 4 13 0 systemd-cgroups [165761.192183] [67649] 0 67649 79 0 3 5 0 systemd-cgroups [165761.192184] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165761.192186] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165761.192187] [67652] 0 67652 610 26 5 9 0 systemd-cgroups [165761.192188] [67654] 0 67654 610 26 4 10 0 systemd-cgroups [165761.192189] Out of memory: Kill process 2435 (irqbalance) score 0 or sacrifice child [165761.192191] Killed process 2435 (irqbalance) total-vm:21844kB, anon-rss:0kB, file-rss:724kB, shmem-rss:0kB [165761.193779] dbus-daemon invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=-900 [165761.193781] dbus-daemon cpuset=/ mems_allowed=0-1 [165761.193782] CPU: 6 PID: 2444 Comm: dbus-daemon Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.193783] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.193784] ffff88082ce3eeb0 00000000e73124d4 ffff880826e8f988 ffffffff816a3e2d [165761.193785] ffff880826e8fa18 ffffffff8169f246 ffffffff810e93ac ffff8808136af0c0 [165761.193786] ffff8808136af0d8 0000000000000202 0000000000000202 ffff880826e8fa08 [165761.193787] Call Trace: [165761.193790] [] dump_stack+0x19/0x1b [165761.193791] [] dump_header+0x90/0x229 [165761.193792] [] ? ktime_get_ts64+0x4c/0xf0 [165761.193793] [] ? delayacct_end+0x8f/0xb0 [165761.193795] [] oom_kill_process+0x254/0x3d0 [165761.193797] [] ? oom_unkillable_task+0xcd/0x120 [165761.193798] [] ? find_lock_task_mm+0x56/0xc0 [165761.193800] [] out_of_memory+0x4b6/0x4f0 [165761.193801] [] __alloc_pages_slowpath+0x5d6/0x724 [165761.193802] [] __alloc_pages_nodemask+0x405/0x420 [165761.193804] [] alloc_pages_vma+0xb5/0x200 [165761.193805] [] ? end_swap_bio_write+0x80/0x80 [165761.193807] [] read_swap_cache_async+0xed/0x160 [165761.193808] [] swapin_readahead+0xa8/0x110 [165761.193809] [] handle_mm_fault+0xadb/0xfa0 [165761.193811] [] __do_page_fault+0x154/0x450 [165761.193812] [] do_page_fault+0x35/0x90 [165761.193813] [] page_fault+0x28/0x30 [165761.193814] Mem-Info: [165761.193818] active_anon:1 inactive_anon:13 isolated_anon:0 active_file:225 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12083 slab_unreclaimable:560256 mapped:1548 shmem:0 pagetables:519 bounce:0 free:39267 free_pcp:82 free_cma:0 [165761.193822] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.193823] lowmem_reserve[]: 0 2580 15620 15620 [165761.193827] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194180kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4416 all_unreclaimable? yes [165761.193828] lowmem_reserve[]: 0 0 13040 13040 [165761.193832] Node 0 Normal free:36684kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:48kB active_file:648kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089612kB kernel_stack:14112kB pagetables:376kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:45997 all_unreclaimable? yes [165761.193833] lowmem_reserve[]: 0 0 0 0 [165761.193837] Node 1 Normal free:45336kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957232kB kernel_stack:16624kB pagetables:1676kB unstable:0kB bounce:0kB free_pcp:328kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1325 all_unreclaimable? yes [165761.193838] lowmem_reserve[]: 0 0 0 0 [165761.193843] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.193847] Node 0 DMA32: 48*4kB (UEM) 1861*8kB (UEM) 1334*16kB (UEM) 503*32kB (UEM) 103*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165761.193852] Node 0 Normal: 155*4kB (UEM) 166*8kB (UEM) 122*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36924kB [165761.193856] Node 1 Normal: 574*4kB (UEM) 410*8kB (UEM) 238*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45256kB [165761.193857] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.193857] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.193858] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.193858] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.193859] 1797 total pagecache pages [165761.193859] 1 pages in swap cache [165761.193860] Swap cache stats: add 314690, delete 314689, find 93727/151345 [165761.193860] Free swap = 16275612kB [165761.193860] Total swap = 16319484kB [165761.193861] 8369063 pages RAM [165761.193861] 0 pages HighMem/MovableOnly [165761.193862] 241261 pages reserved [165761.193862] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165761.193869] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165761.193870] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165761.193871] [ 1046] 0 1046 1588718 3684 132 0 -1000 multipathd [165761.193878] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165761.193879] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165761.193880] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165761.193882] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165761.193883] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165761.193884] [ 3703] 996 3703 24211 161 21 139 0 munged [165761.193885] [ 3951] 0 3951 6464 151 19 62 0 atd [165761.193886] [ 3970] 0 3970 27511 156 12 32 0 agetty [165761.193887] [ 3971] 0 3971 27511 160 10 31 0 agetty [165761.193914] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165761.193915] [67641] 0 67641 79 0 4 28 0 systemd-cgroups [165761.193916] [67646] 0 67646 610 26 4 10 0 systemd-cgroups [165761.193918] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165761.193919] [67648] 0 67648 610 26 4 13 0 systemd-cgroups [165761.193920] [67649] 0 67649 79 0 3 5 0 systemd-cgroups [165761.193921] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165761.193922] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165761.193923] [67652] 0 67652 610 26 5 9 0 systemd-cgroups [165761.193924] [67654] 0 67654 610 26 4 10 0 systemd-cgroups [165761.193925] Out of memory: Kill process 3703 (munged) score 0 or sacrifice child [165761.193926] Killed process 3703 (munged) total-vm:96844kB, anon-rss:0kB, file-rss:644kB, shmem-rss:0kB [165761.211635] dbus-daemon invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=-900 [165761.211637] dbus-daemon cpuset=/ mems_allowed=0-1 [165761.211638] CPU: 6 PID: 2444 Comm: dbus-daemon Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.211639] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.211641] ffff88082ce3eeb0 00000000e73124d4 ffff880826e8f988 ffffffff816a3e2d [165761.211642] ffff880826e8fa18 ffffffff8169f246 ffffffff810e93ac ffff8808136af0c0 [165761.211644] ffff8808136af0d8 0000000000000202 0000000000000202 ffff880826e8fa08 [165761.211644] Call Trace: [165761.211647] [] dump_stack+0x19/0x1b [165761.211648] [] dump_header+0x90/0x229 [165761.211650] [] ? ktime_get_ts64+0x4c/0xf0 [165761.211651] [] ? delayacct_end+0x8f/0xb0 [165761.211654] [] oom_kill_process+0x254/0x3d0 [165761.211655] [] ? oom_unkillable_task+0xcd/0x120 [165761.211657] [] ? find_lock_task_mm+0x56/0xc0 [165761.211659] [] out_of_memory+0x4b6/0x4f0 [165761.211661] [] __alloc_pages_slowpath+0x5d6/0x724 [165761.211662] [] __alloc_pages_nodemask+0x405/0x420 [165761.211664] [] alloc_pages_vma+0xb5/0x200 [165761.211666] [] ? end_swap_bio_write+0x80/0x80 [165761.211667] [] read_swap_cache_async+0xed/0x160 [165761.211669] [] swapin_readahead+0xa8/0x110 [165761.211671] [] handle_mm_fault+0xadb/0xfa0 [165761.211673] [] __do_page_fault+0x154/0x450 [165761.211675] [] do_page_fault+0x35/0x90 [165761.211677] [] page_fault+0x28/0x30 [165761.211677] Mem-Info: [165761.211683] active_anon:1 inactive_anon:13 isolated_anon:0 active_file:225 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12083 slab_unreclaimable:560256 mapped:1548 shmem:0 pagetables:519 bounce:0 free:39267 free_pcp:233 free_cma:0 [165761.211688] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.211689] lowmem_reserve[]: 0 2580 15620 15620 [165761.211694] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194180kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4632 all_unreclaimable? yes [165761.211695] lowmem_reserve[]: 0 0 13040 13040 [165761.211699] Node 0 Normal free:36684kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:48kB active_file:648kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089612kB kernel_stack:14112kB pagetables:376kB unstable:0kB bounce:0kB free_pcp:128kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:46017 all_unreclaimable? yes [165761.211700] lowmem_reserve[]: 0 0 0 0 [165761.211705] Node 1 Normal free:45336kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957232kB kernel_stack:16624kB pagetables:1676kB unstable:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1540 all_unreclaimable? yes [165761.211706] lowmem_reserve[]: 0 0 0 0 [165761.211713] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.211720] Node 0 DMA32: 48*4kB (UEM) 1861*8kB (UEM) 1334*16kB (UEM) 503*32kB (UEM) 103*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165761.211726] Node 0 Normal: 124*4kB (UEM) 166*8kB (UEM) 122*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36800kB [165761.211733] Node 1 Normal: 481*4kB (UEM) 410*8kB (UEM) 238*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44884kB [165761.211734] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.211734] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.211735] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.211736] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.211737] 1797 total pagecache pages [165761.211737] 1 pages in swap cache [165761.211738] Swap cache stats: add 314698, delete 314697, find 93731/151353 [165761.211739] Free swap = 16276128kB [165761.211739] Total swap = 16319484kB [165761.211740] 8369063 pages RAM [165761.211741] 0 pages HighMem/MovableOnly [165761.211741] 241261 pages reserved [165761.211742] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165761.211749] [ 1019] 0 1019 29712 215 28 86 0 lvmetad [165761.211750] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165761.211752] [ 1046] 0 1046 1588718 3684 132 0 -1000 multipathd [165761.211759] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165761.211761] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165761.211763] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165761.211766] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165761.211767] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165761.211769] [ 3951] 0 3951 6464 151 19 62 0 atd [165761.211770] [ 3970] 0 3970 27511 156 12 32 0 agetty [165761.211771] [ 3971] 0 3971 27511 160 10 31 0 agetty [165761.211798] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165761.211800] [67641] 0 67641 79 0 4 28 0 systemd-cgroups [165761.211801] [67646] 0 67646 610 26 4 10 0 systemd-cgroups [165761.211803] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165761.211804] [67648] 0 67648 610 26 4 13 0 systemd-cgroups [165761.211806] [67649] 0 67649 79 0 3 5 0 systemd-cgroups [165761.211807] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165761.211808] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165761.211810] [67652] 0 67652 610 26 5 9 0 systemd-cgroups [165761.211811] [67654] 0 67654 610 26 4 10 0 systemd-cgroups [165761.211812] Out of memory: Kill process 1019 (lvmetad) score 0 or sacrifice child [165761.211813] Killed process 1019 (lvmetad) total-vm:118848kB, anon-rss:0kB, file-rss:860kB, shmem-rss:0kB [165761.219691] kthreadd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 [165761.219693] kthreadd cpuset=/ mems_allowed=0-1 [165761.219694] CPU: 2 PID: 2 Comm: kthreadd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.219695] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.219698] ffff88017a1d0fd0 000000004b8de08d ffff88017a1dfa20 ffffffff816a3e2d [165761.219699] ffff88017a1dfab0 ffffffff8169f246 ffffffff810e93ac ffff88017a127210 [165761.219700] ffff88017a127228 0000000000000206 0000000000000206 ffff88017a1dfaa0 [165761.219701] Call Trace: [165761.219703] [] dump_stack+0x19/0x1b [165761.219705] [] dump_header+0x90/0x229 [165761.219706] [] ? ktime_get_ts64+0x4c/0xf0 [165761.219707] [] ? delayacct_end+0x8f/0xb0 [165761.219709] [] oom_kill_process+0x254/0x3d0 [165761.219711] [] ? oom_unkillable_task+0xcd/0x120 [165761.219712] [] ? find_lock_task_mm+0x56/0xc0 [165761.219714] [] out_of_memory+0x4b6/0x4f0 [165761.219715] [] __alloc_pages_slowpath+0x5d6/0x724 [165761.219716] [] __alloc_pages_nodemask+0x405/0x420 [165761.219719] [] ? insert_kthread_work+0x40/0x40 [165761.219722] [] copy_process+0x1dd/0x1970 [165761.219724] [] ? __switch_to+0xd7/0x510 [165761.219726] [] do_fork+0x91/0x320 [165761.219728] [] ? __schedule+0x39d/0x8b0 [165761.219730] [] kernel_thread+0x26/0x30 [165761.219731] [] kthreadd+0x2c1/0x300 [165761.219733] [] ? kthread_create_on_cpu+0x60/0x60 [165761.219735] [] ret_from_fork+0x58/0x90 [165761.219737] [] ? kthread_create_on_cpu+0x60/0x60 [165761.219738] Mem-Info: [165761.219744] active_anon:1 inactive_anon:13 isolated_anon:0 active_file:225 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12083 slab_unreclaimable:560256 mapped:1548 shmem:0 pagetables:519 bounce:0 free:39267 free_pcp:330 free_cma:0 [165761.219748] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.219749] lowmem_reserve[]: 0 2580 15620 15620 [165761.219753] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194180kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4720 all_unreclaimable? yes [165761.219754] lowmem_reserve[]: 0 0 13040 13040 [165761.219758] Node 0 Normal free:36684kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:48kB active_file:648kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089612kB kernel_stack:14112kB pagetables:376kB unstable:0kB bounce:0kB free_pcp:132kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:46017 all_unreclaimable? yes [165761.219758] lowmem_reserve[]: 0 0 0 0 [165761.219764] Node 1 Normal free:45336kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957232kB kernel_stack:16624kB pagetables:1676kB unstable:0kB bounce:0kB free_pcp:1188kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1668 all_unreclaimable? yes [165761.219765] lowmem_reserve[]: 0 0 0 0 [165761.219773] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.219780] Node 0 DMA32: 48*4kB (UEM) 1861*8kB (UEM) 1334*16kB (UEM) 503*32kB (UEM) 103*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165761.219788] Node 0 Normal: 124*4kB (UEM) 166*8kB (UEM) 122*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36800kB [165761.219792] Node 1 Normal: 435*4kB (UEM) 402*8kB (UEM) 238*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44636kB [165761.219793] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.219794] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.219794] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.219795] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.219795] 1797 total pagecache pages [165761.219796] 1 pages in swap cache [165761.219796] Swap cache stats: add 314706, delete 314705, find 93732/151355 [165761.219797] Free swap = 16276472kB [165761.219797] Total swap = 16319484kB [165761.219798] 8369063 pages RAM [165761.219798] 0 pages HighMem/MovableOnly [165761.219799] 241261 pages reserved [165761.219799] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165761.219807] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165761.219809] [ 1046] 0 1046 1588718 3684 132 0 -1000 multipathd [165761.219818] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165761.219820] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165761.219821] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165761.219825] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165761.219826] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165761.219827] [ 3951] 0 3951 6464 151 19 62 0 atd [165761.219828] [ 3970] 0 3970 27511 156 12 32 0 agetty [165761.219829] [ 3971] 0 3971 27511 160 10 31 0 agetty [165761.219857] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165761.219859] [67641] 0 67641 79 0 4 28 0 systemd-cgroups [165761.219861] [67646] 0 67646 610 26 4 10 0 systemd-cgroups [165761.219862] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165761.219864] [67648] 0 67648 610 26 4 13 0 systemd-cgroups [165761.219866] [67649] 0 67649 79 0 3 5 0 systemd-cgroups [165761.219867] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165761.219868] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165761.219869] [67652] 0 67652 610 26 5 9 0 systemd-cgroups [165761.219870] [67654] 0 67654 610 26 4 10 0 systemd-cgroups [165761.219871] Out of memory: Kill process 3951 (atd) score 0 or sacrifice child [165761.219872] Killed process 3951 (atd) total-vm:25856kB, anon-rss:0kB, file-rss:604kB, shmem-rss:0kB [165761.230582] kthreadd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 [165761.230583] kthreadd cpuset=/ mems_allowed=0-1 [165761.230585] CPU: 2 PID: 2 Comm: kthreadd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.230586] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.230588] ffff88017a1d0fd0 000000004b8de08d ffff88017a1dfa20 ffffffff816a3e2d [165761.230589] ffff88017a1dfab0 ffffffff8169f246 ffffffff810e93ac ffff88017a127210 [165761.230591] ffff88017a127228 0000000000000202 0000000000000202 ffff88017a1dfaa0 [165761.230591] Call Trace: [165761.230594] [] dump_stack+0x19/0x1b [165761.230596] [] dump_header+0x90/0x229 [165761.230598] [] ? ktime_get_ts64+0x4c/0xf0 [165761.230599] [] ? delayacct_end+0x8f/0xb0 [165761.230602] [] oom_kill_process+0x254/0x3d0 [165761.230604] [] ? oom_unkillable_task+0xcd/0x120 [165761.230606] [] ? find_lock_task_mm+0x56/0xc0 [165761.230608] [] out_of_memory+0x4b6/0x4f0 [165761.230610] [] __alloc_pages_slowpath+0x5d6/0x724 [165761.230611] [] __alloc_pages_nodemask+0x405/0x420 [165761.230613] [] ? insert_kthread_work+0x40/0x40 [165761.230614] [] copy_process+0x1dd/0x1970 [165761.230615] [] ? __switch_to+0xd7/0x510 [165761.230616] [] do_fork+0x91/0x320 [165761.230618] [] ? __schedule+0x39d/0x8b0 [165761.230619] [] kernel_thread+0x26/0x30 [165761.230620] [] kthreadd+0x2c1/0x300 [165761.230621] [] ? kthread_create_on_cpu+0x60/0x60 [165761.230623] [] ret_from_fork+0x58/0x90 [165761.230624] [] ? kthread_create_on_cpu+0x60/0x60 [165761.230625] Mem-Info: [165761.230631] active_anon:1 inactive_anon:13 isolated_anon:0 active_file:225 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12083 slab_unreclaimable:560256 mapped:1548 shmem:0 pagetables:519 bounce:0 free:39267 free_pcp:417 free_cma:0 [165761.230637] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.230639] lowmem_reserve[]: 0 2580 15620 15620 [165761.230644] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194180kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4840 all_unreclaimable? yes [165761.230646] lowmem_reserve[]: 0 0 13040 13040 [165761.230652] Node 0 Normal free:36684kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:48kB active_file:648kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089612kB kernel_stack:14112kB pagetables:376kB unstable:0kB bounce:0kB free_pcp:380kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:46058 all_unreclaimable? yes [165761.230653] lowmem_reserve[]: 0 0 0 0 [165761.230656] Node 1 Normal free:45336kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957232kB kernel_stack:16624kB pagetables:1676kB unstable:0kB bounce:0kB free_pcp:1288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1668 all_unreclaimable? yes [165761.230657] lowmem_reserve[]: 0 0 0 0 [165761.230664] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.230672] Node 0 DMA32: 48*4kB (UEM) 1861*8kB (UEM) 1334*16kB (UEM) 503*32kB (UEM) 103*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165761.230678] Node 0 Normal: 64*4kB (EM) 165*8kB (UEM) 122*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36552kB [165761.230686] Node 1 Normal: 435*4kB (UEM) 402*8kB (UEM) 238*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44636kB [165761.230687] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.230687] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.230688] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.230689] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.230689] 1797 total pagecache pages [165761.230689] 1 pages in swap cache [165761.230690] Swap cache stats: add 314714, delete 314713, find 93733/151357 [165761.230691] Free swap = 16276684kB [165761.230691] Total swap = 16319484kB [165761.230692] 8369063 pages RAM [165761.230692] 0 pages HighMem/MovableOnly [165761.230692] 241261 pages reserved [165761.230692] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165761.230700] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165761.230702] [ 1046] 0 1046 1588718 3684 132 0 -1000 multipathd [165761.230710] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165761.230712] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165761.230714] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165761.230717] [ 3687] 0 3687 6779 148 18 63 0 xinetd [165761.230718] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165761.230720] [ 3970] 0 3970 27511 156 12 32 0 agetty [165761.230722] [ 3971] 0 3971 27511 160 10 31 0 agetty [165761.230749] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165761.230751] [67641] 0 67641 79 0 4 28 0 systemd-cgroups [165761.230752] [67646] 0 67646 610 26 4 10 0 systemd-cgroups [165761.230754] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165761.230756] [67648] 0 67648 610 26 4 13 0 systemd-cgroups [165761.230757] [67649] 0 67649 79 0 3 5 0 systemd-cgroups [165761.230759] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165761.230760] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165761.230762] [67652] 0 67652 610 26 5 9 0 systemd-cgroups [165761.230763] [67654] 0 67654 610 26 4 10 0 systemd-cgroups [165761.230765] Out of memory: Kill process 3687 (xinetd) score 0 or sacrifice child [165761.230766] Killed process 3687 (xinetd) total-vm:27116kB, anon-rss:0kB, file-rss:592kB, shmem-rss:0kB [165761.244942] kworker/u481:2 invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 [165761.244943] kworker/u481:2 cpuset=/ mems_allowed=0-1 [165761.244945] CPU: 16 PID: 67631 Comm: kworker/u481:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.244946] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.244949] Workqueue: events_unbound __call_usermodehelper [165761.244951] ffff880417b11fa0 0000000052588492 ffff8807e9bdb9a0 ffffffff816a3e2d [165761.244952] ffff8807e9bdba30 ffffffff8169f246 ffffffff810e93ac ffff880036a28c90 [165761.244954] ffff880036a28ca8 0000000000000202 0000000000000202 ffff8807e9bdba20 [165761.244954] Call Trace: [165761.244957] [] dump_stack+0x19/0x1b [165761.244958] [] dump_header+0x90/0x229 [165761.244960] [] ? ktime_get_ts64+0x4c/0xf0 [165761.244961] [] ? delayacct_end+0x8f/0xb0 [165761.244964] [] oom_kill_process+0x254/0x3d0 [165761.244965] [] ? oom_unkillable_task+0xcd/0x120 [165761.244967] [] ? find_lock_task_mm+0x56/0xc0 [165761.244969] [] out_of_memory+0x4b6/0x4f0 [165761.244971] [] __alloc_pages_slowpath+0x5d6/0x724 [165761.244972] [] __alloc_pages_nodemask+0x405/0x420 [165761.244974] [] ? ____call_usermodehelper+0x130/0x130 [165761.244976] [] copy_process+0x1dd/0x1970 [165761.244978] [] ? account_entity_dequeue+0xae/0xd0 [165761.244980] [] do_fork+0x91/0x320 [165761.244982] [] ? finish_task_switch+0x54/0x160 [165761.244984] [] kernel_thread+0x26/0x30 [165761.244985] [] __call_usermodehelper+0x2e/0x90 [165761.244987] [] process_one_work+0x17a/0x440 [165761.244989] [] worker_thread+0x126/0x3c0 [165761.244990] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165761.244992] [] kthread+0xcf/0xe0 [165761.244993] [] ? insert_kthread_work+0x40/0x40 [165761.244995] [] ret_from_fork+0x58/0x90 [165761.244996] [] ? insert_kthread_work+0x40/0x40 [165761.244996] Mem-Info: [165761.245002] active_anon:1 inactive_anon:13 isolated_anon:0 active_file:142 inactive_file:14 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12083 slab_unreclaimable:560256 mapped:1548 shmem:0 pagetables:519 bounce:0 free:39267 free_pcp:0 free_cma:0 [165761.245006] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.245008] lowmem_reserve[]: 0 2580 15620 15620 [165761.245013] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:344kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194180kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:86 all_unreclaimable? yes [165761.245014] lowmem_reserve[]: 0 0 13040 13040 [165761.245019] Node 0 Normal free:36684kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:48kB active_file:316kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19400kB slab_unreclaimable:1089612kB kernel_stack:14112kB pagetables:376kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:734 all_unreclaimable? yes [165761.245020] lowmem_reserve[]: 0 0 0 0 [165761.245024] Node 1 Normal free:45336kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:236kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957232kB kernel_stack:16624kB pagetables:1676kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1240 all_unreclaimable? yes [165761.245025] lowmem_reserve[]: 0 0 0 0 [165761.245032] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.245038] Node 0 DMA32: 47*4kB (UEM) 1861*8kB (UEM) 1333*16kB (UEM) 502*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59508kB [165761.245043] Node 0 Normal: 177*4kB (UEM) 176*8kB (UEM) 122*16kB (UEM) 214*32kB (UEM) 385*64kB (UEM) 10*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37092kB [165761.245049] Node 1 Normal: 713*4kB (UEM) 429*8kB (UEM) 241*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46012kB [165761.245050] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.245051] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.245052] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.245052] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.245053] 1797 total pagecache pages [165761.245053] 0 pages in swap cache [165761.245054] Swap cache stats: add 314724, delete 314724, find 93734/151360 [165761.245054] Free swap = 16276936kB [165761.245055] Total swap = 16319484kB [165761.245056] 8369063 pages RAM [165761.245056] 0 pages HighMem/MovableOnly [165761.245056] 241261 pages reserved [165761.245057] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165761.245064] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165761.245066] [ 1046] 0 1046 1588718 3684 132 0 -1000 multipathd [165761.245073] [ 2373] 0 2373 13863 114 27 119 -1000 auditd [165761.245076] [ 2444] 81 2444 9196 201 21 148 -900 dbus-daemon [165761.245078] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165761.245081] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165761.245083] [ 3970] 0 3970 27511 156 12 32 0 agetty [165761.245084] [ 3971] 0 3971 27511 160 10 31 0 agetty [165761.245109] [67635] 0 67635 90 13 4 13 0 systemd-cgroups [165761.245110] [67641] 0 67641 79 0 4 28 0 systemd-cgroups [165761.245112] [67646] 0 67646 610 26 4 10 0 systemd-cgroups [165761.245113] [67647] 0 67647 79 0 4 5 0 systemd-cgroups [165761.245115] [67648] 0 67648 610 26 4 13 0 systemd-cgroups [165761.245117] [67649] 0 67649 79 0 3 5 0 systemd-cgroups [165761.245118] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165761.245119] [67651] 0 67651 79 0 3 4 0 systemd-cgroups [165761.245121] [67652] 0 67652 610 26 5 9 0 systemd-cgroups [165761.245123] [67654] 0 67654 610 26 4 10 0 systemd-cgroups [165761.245124] Out of memory: Kill process 3971 (agetty) score 0 or sacrifice child [165761.245125] Killed process 3971 (agetty) total-vm:110044kB, anon-rss:0kB, file-rss:640kB, shmem-rss:0kB [165761.362541] kworker/u480:2: page allocation failure: order:8, mode:0x80d0 [165761.362543] CPU: 0 PID: 67636 Comm: kworker/u480:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.362543] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.362550] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165761.362551] 00000000000080d0 00000000987a9902 ffff88052cdf7810 ffffffff816a3e2d [165761.362552] ffff88052cdf78a0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165761.362553] ffffffffffffff00 000080d000000000 ffff88052cdf7870 00000000987a9902 [165761.362554] Call Trace: [165761.362556] [] dump_stack+0x19/0x1b [165761.362558] [] warn_alloc_failed+0x110/0x180 [165761.362560] [] ? drain_pages+0xb0/0xb0 [165761.362561] [] __alloc_pages_slowpath+0x6b6/0x724 [165761.362563] [] __alloc_pages_nodemask+0x405/0x420 [165761.362564] [] dma_generic_alloc_coherent+0x8f/0x140 [165761.362567] [] x86_swiotlb_alloc_coherent+0x21/0x50 [165761.362582] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165761.362587] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165761.362593] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165761.362595] [] ? dma_pool_free+0xa7/0xd0 [165761.362600] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165761.362603] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165761.362608] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165761.362615] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165761.362617] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165761.362623] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165761.362628] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165761.362631] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165761.362632] [] process_one_work+0x17a/0x440 [165761.362633] [] worker_thread+0x126/0x3c0 [165761.362635] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165761.362636] [] kthread+0xcf/0xe0 [165761.362637] [] ? insert_kthread_work+0x40/0x40 [165761.362638] [] ret_from_fork+0x58/0x90 [165761.362639] [] ? insert_kthread_work+0x40/0x40 [165761.362640] Mem-Info: [165761.362644] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:289 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560212 mapped:1547 shmem:0 pagetables:400 bounce:0 free:39499 free_pcp:86 free_cma:0 [165761.362648] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.362649] lowmem_reserve[]: 0 2580 15620 15620 [165761.362653] Node 0 DMA32 free:59540kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194172kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:22 all_unreclaimable? no [165761.362654] lowmem_reserve[]: 0 0 13040 13040 [165761.362658] Node 0 Normal free:37036kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1036kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089516kB kernel_stack:14112kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:236kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:143 all_unreclaimable? no [165761.362659] lowmem_reserve[]: 0 0 0 0 [165761.362663] Node 1 Normal free:45840kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:108kB inactive_file:128kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957160kB kernel_stack:16624kB pagetables:1336kB unstable:0kB bounce:0kB free_pcp:108kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:271 all_unreclaimable? no [165761.362664] lowmem_reserve[]: 0 0 0 0 [165761.362669] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.362674] Node 0 DMA32: 62*4kB (UEM) 1862*8kB (UEM) 1334*16kB (UEM) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59560kB [165761.362679] Node 0 Normal: 121*4kB (UEM) 182*8kB (UEM) 126*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37204kB [165761.362683] Node 1 Normal: 600*4kB (UEM) 429*8kB (UEM) 240*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45832kB [165761.362684] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.362685] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.362685] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.362686] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.362686] 1779 total pagecache pages [165761.362687] 36 pages in swap cache [165761.362688] Swap cache stats: add 314796, delete 314760, find 93745/151382 [165761.362688] Free swap = 16277040kB [165761.362688] Total swap = 16319484kB [165761.362689] 8369063 pages RAM [165761.362689] 0 pages HighMem/MovableOnly [165761.362689] 241261 pages reserved [165761.362850] kworker/u480:2: page allocation failure: order:8, mode:0x80d0 [165761.362851] CPU: 0 PID: 67636 Comm: kworker/u480:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165761.362851] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165761.362854] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165761.362855] 00000000000080d0 00000000987a9902 ffff88052cdf77c0 ffffffff816a3e2d [165761.362856] ffff88052cdf7850 ffffffff81188820 ffffffff8118b540 0000000000000000 [165761.362857] ffffffffffffff00 000080d000000000 ffff88052cdf7820 00000000987a9902 [165761.362857] Call Trace: [165761.362859] [] dump_stack+0x19/0x1b [165761.362861] [] warn_alloc_failed+0x110/0x180 [165761.362862] [] ? drain_pages+0xb0/0xb0 [165761.362863] [] __alloc_pages_slowpath+0x6b6/0x724 [165761.362864] [] __alloc_pages_nodemask+0x405/0x420 [165761.362866] [] alloc_pages_current+0x98/0x110 [165761.362867] [] __get_free_pages+0xe/0x40 [165761.362870] [] swiotlb_alloc_coherent+0x5e/0x150 [165761.362871] [] x86_swiotlb_alloc_coherent+0x41/0x50 [165761.362877] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [165761.362882] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [165761.362887] [] create_qp_common+0x536/0x1000 [mlx4_ib] [165761.362889] [] ? dma_pool_free+0xa7/0xd0 [165761.362893] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165761.362897] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165761.362901] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165761.362913] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165761.362915] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165761.362918] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165761.362922] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165761.362924] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165761.362925] [] process_one_work+0x17a/0x440 [165761.362926] [] worker_thread+0x126/0x3c0 [165761.362928] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165761.362929] [] kthread+0xcf/0xe0 [165761.362930] [] ? insert_kthread_work+0x40/0x40 [165761.362931] [] ret_from_fork+0x58/0x90 [165761.362932] [] ? insert_kthread_work+0x40/0x40 [165761.362932] Mem-Info: [165761.362937] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:289 inactive_file:0 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560212 mapped:1547 shmem:0 pagetables:400 bounce:0 free:39499 free_pcp:44 free_cma:0 [165761.362940] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165761.362941] lowmem_reserve[]: 0 2580 15620 15620 [165761.362945] Node 0 DMA32 free:59540kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194172kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:22 all_unreclaimable? no [165761.362946] lowmem_reserve[]: 0 0 13040 13040 [165761.362950] Node 0 Normal free:37036kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:1036kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089516kB kernel_stack:14112kB pagetables:240kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165761.362951] lowmem_reserve[]: 0 0 0 0 [165761.362954] Node 1 Normal free:45840kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:108kB inactive_file:128kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957160kB kernel_stack:16624kB pagetables:1336kB unstable:0kB bounce:0kB free_pcp:176kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [165761.362955] lowmem_reserve[]: 0 0 0 0 [165761.362961] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165761.362965] Node 0 DMA32: 62*4kB (UEM) 1862*8kB (UEM) 1334*16kB (UEM) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59560kB [165761.362969] Node 0 Normal: 116*4kB (UEM) 182*8kB (UEM) 126*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37184kB [165761.362973] Node 1 Normal: 527*4kB (EM) 426*8kB (UEM) 240*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45516kB [165761.362974] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.362975] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.362976] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165761.362976] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165761.362976] 1779 total pagecache pages [165761.362977] 37 pages in swap cache [165761.362978] Swap cache stats: add 314801, delete 314764, find 93752/151390 [165761.362978] Free swap = 16277056kB [165761.362978] Total swap = 16319484kB [165761.362979] 8369063 pages RAM [165761.362979] 0 pages HighMem/MovableOnly [165761.362979] 241261 pages reserved [165782.855814] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165782.885075] systemd-cgroups cpuset=/ mems_allowed=0-1 [165782.903616] CPU: 15 PID: 67651 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165782.945055] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165782.983626] ffff8806a57deeb0 000000003cffd884 ffff88042b12b658 ffffffff816a3e2d [165783.009614] ffff88042b12b6e8 ffffffff8169f246 ffffffff810e93ac ffff880036a28980 [165783.035587] ffff880036a28998 0000000000000202 0000000000000202 ffff88042b12b6d8 [165783.061566] Call Trace: [165783.071193] [] dump_stack+0x19/0x1b [165783.089674] [] dump_header+0x90/0x229 [165783.108673] [] ? ktime_get_ts64+0x4c/0xf0 [165783.128763] [] ? delayacct_end+0x8f/0xb0 [165783.148595] [] oom_kill_process+0x254/0x3d0 [165783.169113] [] ? oom_unkillable_task+0xcd/0x120 [165783.190830] [] ? find_lock_task_mm+0x56/0xc0 [165783.211701] [] out_of_memory+0x4b6/0x4f0 [165783.231481] [] __alloc_pages_slowpath+0x5d6/0x724 [165783.253965] [] __alloc_pages_nodemask+0x405/0x420 [165783.276162] [] alloc_pages_vma+0xb5/0x200 [165783.296137] [] read_swap_cache_async+0xed/0x160 [165783.317827] [] swapin_readahead+0xa8/0x110 [165783.338136] [] handle_mm_fault+0xadb/0xfa0 [165783.358367] [] __do_page_fault+0x154/0x450 [165783.378424] [] do_page_fault+0x35/0x90 [165783.397339] [] page_fault+0x28/0x30 [165783.415315] [] ? strnlen_user+0x3e/0x100 [165783.434807] [] ? create_elf_tables+0x407/0x523 [165783.456018] [] load_elf_binary+0xa00/0xe00 [165783.476078] [] ? load_elf_library+0x220/0x220 [165783.496950] [] search_binary_handler+0xed/0x300 [165783.518317] [] do_execve_common.isra.25+0x5b6/0x6c0 [165783.540844] [] do_execve+0x18/0x20 [165783.558405] [] ____call_usermodehelper+0xfc/0x130 [165783.580297] [] ? ____call_usermodehelper+0x130/0x130 [165783.603031] [] call_helper+0x1e/0x20 [165783.621282] [] ret_from_fork+0x58/0x90 [165783.640152] [] ? ____call_usermodehelper+0x130/0x130 [165783.662999] Mem-Info: [165783.671489] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:115 inactive_file:120 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560208 mapped:1547 shmem:0 pagetables:351 bounce:0 free:39317 free_pcp:1 free_cma:0 [165783.783670] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165783.923312] lowmem_reserve[]: 0 2580 15620 15620 [165783.939668] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194172kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:224858 all_unreclaimable? yes [165784.091080] lowmem_reserve[]: 0 0 13040 13040 [165784.106566] Node 0 Normal free:36692kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:212kB inactive_file:96kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089516kB kernel_stack:14128kB pagetables:164kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1042 all_unreclaimable? yes [165784.261661] lowmem_reserve[]: 0 0 0 0 [165784.274987] Node 1 Normal free:45564kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:368kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957144kB kernel_stack:16656kB pagetables:1216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:904 all_unreclaimable? yes [165784.431337] lowmem_reserve[]: 0 0 0 0 [165784.444771] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165784.491619] Node 0 DMA32: 42*4kB (UEM) 1860*8kB (UEM) 1332*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59432kB [165784.542846] Node 0 Normal: 127*4kB (UEM) 143*8kB (EM) 112*16kB (EM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36692kB [165784.595687] Node 1 Normal: 559*4kB (UEM) 420*8kB (UEM) 240*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45564kB [165784.648069] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165784.678309] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165784.707829] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165784.738155] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165784.767526] 1780 total pagecache pages [165784.781618] 0 pages in swap cache [165784.794253] Swap cache stats: add 314825, delete 314825, find 93770/151418 [165784.818579] Free swap = 16277028kB [165784.831666] Total swap = 16319484kB [165784.844731] 8369063 pages RAM [165784.856035] 0 pages HighMem/MovableOnly [165784.870280] 241261 pages reserved [165784.882879] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165784.910425] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165784.940375] [ 1046] 0 1046 1588718 3700 132 0 -1000 multipathd [165784.969553] [ 2373] 0 2373 13863 113 27 120 -1000 auditd [165784.997504] [ 2444] 81 2444 9196 200 21 149 -900 dbus-daemon [165785.026794] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165785.054822] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165785.082297] [ 3970] 0 3970 27511 156 12 32 0 agetty [165785.110124] [67635] 0 67635 90 11 4 15 0 systemd-cgroups [165785.140549] [67641] 0 67641 79 8 4 7 0 systemd-cgroups [165785.170965] [67646] 0 67646 610 26 4 13 0 systemd-cgroups [165785.201494] [67647] 0 67647 79 0 4 6 0 systemd-cgroups [165785.231983] [67648] 0 67648 610 26 4 16 0 systemd-cgroups [165785.262364] [67649] 0 67649 79 0 4 11 0 systemd-cgroups [165785.292789] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165785.323114] [67651] 0 67651 79 0 3 6 0 systemd-cgroups [165785.353335] [67652] 0 67652 610 26 5 13 0 systemd-cgroups [165785.383608] [67654] 0 67654 610 25 5 25 0 systemd-cgroups [165785.413859] Out of memory: Kill process 3970 (agetty) score 0 or sacrifice child [165785.439521] Killed process 3970 (agetty) total-vm:110044kB, anon-rss:0kB, file-rss:624kB, shmem-rss:0kB [165785.472242] systemd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165785.498363] systemd cpuset=/ mems_allowed=0-1 [165785.513776] CPU: 19 PID: 1 Comm: systemd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165785.549512] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165785.587531] ffff88017a1d0000 00000000f5e675ef ffff88017a1db988 ffffffff816a3e2d [165785.612994] ffff88017a1dba18 ffffffff8169f246 ffffffff810e93ac ffff88017a1271a0 [165785.638442] ffff88017a1271b8 0000000000000202 0000000000000202 ffff88017a1dba08 [165785.663883] Call Trace: [165785.672953] [] dump_stack+0x19/0x1b [165785.690891] [] dump_header+0x90/0x229 [165785.709363] [] ? ktime_get_ts64+0x4c/0xf0 [165785.728956] [] ? delayacct_end+0x8f/0xb0 [165785.748243] [] oom_kill_process+0x254/0x3d0 [165785.768365] [] ? oom_unkillable_task+0xcd/0x120 [165785.789615] [] ? find_lock_task_mm+0x56/0xc0 [165785.809987] [] out_of_memory+0x4b6/0x4f0 [165785.829194] [] __alloc_pages_slowpath+0x5d6/0x724 [165785.850981] [] __alloc_pages_nodemask+0x405/0x420 [165785.872748] [] alloc_pages_vma+0xb5/0x200 [165785.892264] [] read_swap_cache_async+0xed/0x160 [165785.913485] [] swapin_readahead+0xa8/0x110 [165785.933278] [] handle_mm_fault+0xadb/0xfa0 [165785.953062] [] __do_page_fault+0x154/0x450 [165785.972835] [] do_page_fault+0x35/0x90 [165785.991470] [] page_fault+0x28/0x30 [165786.009241] Mem-Info: [165786.017573] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:115 inactive_file:120 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560208 mapped:1547 shmem:0 pagetables:351 bounce:0 free:39317 free_pcp:0 free_cma:0 [165786.129000] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165786.268112] lowmem_reserve[]: 0 2580 15620 15620 [165786.284341] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194172kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:234034 all_unreclaimable? yes [165786.435606] lowmem_reserve[]: 0 0 13040 13040 [165786.451124] Node 0 Normal free:36824kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:52kB inactive_file:256kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089516kB kernel_stack:14080kB pagetables:116kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:539 all_unreclaimable? yes [165786.605827] lowmem_reserve[]: 0 0 0 0 [165786.619110] Node 1 Normal free:45564kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:368kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957144kB kernel_stack:16656kB pagetables:1216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1035 all_unreclaimable? yes [165786.775180] lowmem_reserve[]: 0 0 0 0 [165786.788504] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165786.835085] Node 0 DMA32: 42*4kB (UEM) 1860*8kB (UEM) 1332*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59432kB [165786.886044] Node 0 Normal: 148*4kB (UEM) 143*8kB (EM) 115*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36824kB [165786.938521] Node 1 Normal: 559*4kB (UEM) 420*8kB (UEM) 240*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45564kB [165786.990221] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165787.020267] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165787.049451] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165787.079493] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165787.108668] 1780 total pagecache pages [165787.122390] 0 pages in swap cache [165787.134676] Swap cache stats: add 314825, delete 314825, find 93770/151418 [165787.158709] Free swap = 16277156kB [165787.171586] Total swap = 16319484kB [165787.184466] 8369063 pages RAM [165787.195618] 0 pages HighMem/MovableOnly [165787.209623] 241261 pages reserved [165787.221912] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165787.249105] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165787.278875] [ 1046] 0 1046 1588718 3698 132 0 -1000 multipathd [165787.307789] [ 2373] 0 2373 13863 113 27 120 -1000 auditd [165787.335548] [ 2444] 81 2444 9196 200 21 149 -900 dbus-daemon [165787.364722] [ 2514] 0 2514 1770 27 8 38 0 mcelog [165787.392444] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165787.419608] [67635] 0 67635 90 11 4 15 0 systemd-cgroups [165787.449865] [67641] 0 67641 79 8 4 7 0 systemd-cgroups [165787.480108] [67646] 0 67646 610 26 4 13 0 systemd-cgroups [165787.510341] [67647] 0 67647 79 0 4 6 0 systemd-cgroups [165787.540526] [67648] 0 67648 610 26 4 16 0 systemd-cgroups [165787.570681] [67649] 0 67649 79 0 4 11 0 systemd-cgroups [165787.600819] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165787.630924] [67651] 0 67651 79 0 3 6 0 systemd-cgroups [165787.661029] [67652] 0 67652 610 26 5 13 0 systemd-cgroups [165787.691096] [67654] 0 67654 610 25 5 25 0 systemd-cgroups [165787.721136] Out of memory: Kill process 2514 (mcelog) score 0 or sacrifice child [165787.746586] Killed process 2514 (mcelog) total-vm:7080kB, anon-rss:0kB, file-rss:108kB, shmem-rss:0kB [165787.779304] ll_ost_io01_104 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [165787.807774] ll_ost_io01_104 cpuset=/ mems_allowed=1 [165787.825116] CPU: 11 PID: 226541 Comm: ll_ost_io01_104 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165787.865564] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165787.903785] ffff88081d3a8fd0 00000000c8832be1 ffff8807947af998 ffffffff816a3e2d [165787.929522] ffff8807947afa28 ffffffff8169f246 ffffffff810e93ac ffff8807ebda5de0 [165787.955158] ffff8807ebda5df8 0000000000000206 0000000000000206 ffff8807947afa18 [165787.980783] Call Trace: [165787.989947] [] dump_stack+0x19/0x1b [165788.008105] [] dump_header+0x90/0x229 [165788.026636] [] ? ktime_get_ts64+0x4c/0xf0 [165788.046469] [] ? delayacct_end+0x8f/0xb0 [165788.065987] [] oom_kill_process+0x254/0x3d0 [165788.086338] [] ? oom_unkillable_task+0xcd/0x120 [165788.107830] [] ? find_lock_task_mm+0x56/0xc0 [165788.128342] [] out_of_memory+0x4b6/0x4f0 [165788.147645] [] __alloc_pages_slowpath+0x5d6/0x724 [165788.169536] [] __alloc_pages_nodemask+0x405/0x420 [165788.191388] [] __vmalloc_node_range+0x12f/0x270 [165788.212870] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165788.236575] [] vzalloc_node+0x47/0x50 [165788.255102] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165788.278832] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165788.301982] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [165788.325604] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [165788.347386] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [165788.372890] [] kthread+0xcf/0xe0 [165788.390082] [] ? insert_kthread_work+0x40/0x40 [165788.411152] [] ret_from_fork+0x58/0x90 [165788.429969] [] ? insert_kthread_work+0x40/0x40 [165788.451022] Mem-Info: [165788.459595] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:75 inactive_file:160 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560208 mapped:1547 shmem:0 pagetables:331 bounce:0 free:39288 free_pcp:0 free_cma:0 [165788.571501] Node 1 Normal free:45316kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:368kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957144kB kernel_stack:16656kB pagetables:1188kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1104 all_unreclaimable? yes [165788.727278] lowmem_reserve[]: 0 0 0 0 [165788.740662] Node 1 Normal: 570*4kB (UEM) 421*8kB (UEM) 240*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45616kB [165788.792172] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165788.822043] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165788.851144] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165788.881047] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165788.909947] 1780 total pagecache pages [165788.923421] 0 pages in swap cache [165788.935432] Swap cache stats: add 314827, delete 314827, find 93770/151419 [165788.959138] Free swap = 16277308kB [165788.971716] Total swap = 16319484kB [165788.984279] 8369063 pages RAM [165788.995049] 0 pages HighMem/MovableOnly [165789.008748] 241261 pages reserved [165789.020748] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165789.047616] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165789.076994] [ 1046] 0 1046 1588718 3694 132 0 -1000 multipathd [165789.105734] [ 2373] 0 2373 13863 113 27 120 -1000 auditd [165789.133315] [ 2444] 81 2444 9196 200 21 149 -900 dbus-daemon [165789.162313] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165789.189413] [67635] 0 67635 90 11 4 15 0 systemd-cgroups [165789.219562] [67641] 0 67641 79 8 4 7 0 systemd-cgroups [165789.249637] [67646] 0 67646 610 26 4 13 0 systemd-cgroups [165789.279825] [67647] 0 67647 79 0 4 6 0 systemd-cgroups [165789.309980] [67648] 0 67648 610 26 4 16 0 systemd-cgroups [165789.340046] [67649] 0 67649 79 0 4 11 0 systemd-cgroups [165789.370030] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165789.399960] [67651] 0 67651 79 0 3 6 0 systemd-cgroups [165789.429899] [67652] 0 67652 610 26 5 13 0 systemd-cgroups [165789.459845] [67654] 0 67654 610 25 5 25 0 systemd-cgroups [165789.489770] Out of memory: Kill process 67654 (systemd-cgroups) score 0 or sacrifice child [165789.518063] Killed process 67654 (systemd-cgroups) total-vm:2440kB, anon-rss:0kB, file-rss:100kB, shmem-rss:0kB [165789.553389] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165789.581644] systemd-cgroups cpuset=/ mems_allowed=0-1 [165789.599313] CPU: 29 PID: 67649 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165789.639245] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165789.677376] ffff8805acfabf40 00000000ac06895f ffff88042d8a7988 ffffffff816a3e2d [165789.703032] ffff88042d8a7a18 ffffffff8169f246 ffffffff810e93ac ffff8802b1253b40 [165789.728735] ffff8802b1253b58 0000000000000202 0000000000000202 ffff88042d8a7a08 [165789.754408] Call Trace: [165789.763662] [] dump_stack+0x19/0x1b [165789.781842] [] dump_header+0x90/0x229 [165789.800585] [] ? ktime_get_ts64+0x4c/0xf0 [165789.820302] [] ? delayacct_end+0x8f/0xb0 [165789.839874] [] oom_kill_process+0x254/0x3d0 [165789.860222] [] ? oom_unkillable_task+0xcd/0x120 [165789.881717] [] ? find_lock_task_mm+0x56/0xc0 [165789.902259] [] out_of_memory+0x4b6/0x4f0 [165789.921510] [] __alloc_pages_slowpath+0x5d6/0x724 [165789.943464] [] __alloc_pages_nodemask+0x405/0x420 [165789.965407] [] alloc_pages_vma+0xb5/0x200 [165789.984965] [] ? end_swap_bio_write+0x80/0x80 [165790.005716] [] read_swap_cache_async+0xed/0x160 [165790.027168] [] ? task_numa_fault+0x8d0/0xbb0 [165790.047663] [] swapin_readahead+0xa8/0x110 [165790.067591] [] handle_mm_fault+0xadb/0xfa0 [165790.087507] [] __do_page_fault+0x154/0x450 [165790.107450] [] do_page_fault+0x35/0x90 [165790.126204] [] page_fault+0x28/0x30 [165790.144055] Mem-Info: [165790.152551] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:126 inactive_file:109 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560208 mapped:1547 shmem:0 pagetables:331 bounce:0 free:39364 free_pcp:73 free_cma:0 [165790.264909] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165790.404507] lowmem_reserve[]: 0 2580 15620 15620 [165790.420861] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194172kB kernel_stack:1616kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:250426 all_unreclaimable? yes [165790.572827] lowmem_reserve[]: 0 0 13040 13040 [165790.588459] Node 0 Normal free:36828kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:256kB inactive_file:52kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089516kB kernel_stack:14080kB pagetables:96kB unstable:0kB bounce:0kB free_pcp:32kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:693 all_unreclaimable? yes [165790.743726] lowmem_reserve[]: 0 0 0 0 [165790.757069] Node 1 Normal free:45368kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:368kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957144kB kernel_stack:16656kB pagetables:1184kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:31960 all_unreclaimable? yes [165790.913311] lowmem_reserve[]: 0 0 0 0 [165790.926629] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165790.973206] Node 0 DMA32: 44*4kB (UEM) 1860*8kB (UEM) 1334*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59472kB [165791.024154] Node 0 Normal: 149*4kB (UEM) 147*8kB (UEM) 118*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36908kB [165791.076911] Node 1 Normal: 571*4kB (UEM) 421*8kB (UEM) 240*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45620kB [165791.128602] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165791.158647] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165791.187830] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165791.217875] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165791.247040] 1780 total pagecache pages [165791.260759] 0 pages in swap cache [165791.273042] Swap cache stats: add 314830, delete 314830, find 93770/151419 [165791.297069] Free swap = 16277356kB [165791.309940] Total swap = 16319484kB [165791.322811] 8369063 pages RAM [165791.333975] 0 pages HighMem/MovableOnly [165791.347993] 241261 pages reserved [165791.360282] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165791.387470] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165791.417228] [ 1046] 0 1046 1588718 3692 132 0 -1000 multipathd [165791.446135] [ 2373] 0 2373 13863 113 27 120 -1000 auditd [165791.473890] [ 2444] 81 2444 9196 200 21 149 -900 dbus-daemon [165791.503070] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165791.530258] [67635] 0 67635 90 11 4 15 0 systemd-cgroups [165791.560534] [67641] 0 67641 79 8 4 7 0 systemd-cgroups [165791.590780] [67646] 0 67646 610 26 4 13 0 systemd-cgroups [165791.621006] [67647] 0 67647 79 0 4 6 0 systemd-cgroups [165791.651206] [67648] 0 67648 610 26 4 16 0 systemd-cgroups [165791.681380] [67649] 0 67649 79 0 4 11 0 systemd-cgroups [165791.711540] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165791.741655] [67651] 0 67651 79 0 3 6 0 systemd-cgroups [165791.771744] [67652] 0 67652 610 26 5 13 0 systemd-cgroups [165791.801825] Out of memory: Kill process 67648 (systemd-cgroups) score 0 or sacrifice child [165791.830155] Killed process 67648 (systemd-cgroups) total-vm:2440kB, anon-rss:0kB, file-rss:104kB, shmem-rss:0kB [165791.866030] auditd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=-1000 [165791.893101] auditd cpuset=/ mems_allowed=0-1 [165791.908275] CPU: 1 PID: 2373 Comm: auditd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165791.944340] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165791.982419] ffff88082c1caf70 00000000689d2a54 ffff88082ae7f988 ffffffff816a3e2d [165792.007929] ffff88082ae7fa18 ffffffff8169f246 ffffffff810e93ac ffff8808162f74b0 [165792.033452] ffff8808162f74c8 0000000000000202 0000000000000202 ffff88082ae7fa08 [165792.058969] Call Trace: [165792.068089] [] dump_stack+0x19/0x1b [165792.086063] [] dump_header+0x90/0x229 [165792.104604] [] ? ktime_get_ts64+0x4c/0xf0 [165792.124280] [] ? delayacct_end+0x8f/0xb0 [165792.143648] [] oom_kill_process+0x254/0x3d0 [165792.163849] [] ? oom_unkillable_task+0xcd/0x120 [165792.185173] [] ? find_lock_task_mm+0x56/0xc0 [165792.205619] [] out_of_memory+0x4b6/0x4f0 [165792.224897] [] __alloc_pages_slowpath+0x5d6/0x724 [165792.246731] [] __alloc_pages_nodemask+0x405/0x420 [165792.268542] [] alloc_pages_vma+0xb5/0x200 [165792.288043] [] read_swap_cache_async+0xed/0x160 [165792.309273] [] ? mem_cgroup_update_page_stat+0x16/0x50 [165792.332519] [] swapin_readahead+0xa8/0x110 [165792.352336] [] handle_mm_fault+0xadb/0xfa0 [165792.372140] [] __do_page_fault+0x154/0x450 [165792.391939] [] do_page_fault+0x35/0x90 [165792.410590] [] page_fault+0x28/0x30 [165792.428391] Mem-Info: [165792.436735] active_anon:0 inactive_anon:6 isolated_anon:0 active_file:146 inactive_file:42 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560188 mapped:1551 shmem:0 pagetables:322 bounce:0 free:39333 free_pcp:0 free_cma:0 [165792.548091] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165792.687260] lowmem_reserve[]: 0 2580 15620 15620 [165792.703486] Node 0 DMA32 free:59472kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1616kB pagetables:20kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1268 all_unreclaimable? yes [165792.853459] lowmem_reserve[]: 0 0 13040 13040 [165792.868964] Node 0 Normal free:36908kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:336kB inactive_file:0kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14080kB pagetables:84kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:594 all_unreclaimable? yes [165793.023153] lowmem_reserve[]: 0 0 0 0 [165793.036441] Node 1 Normal free:45372kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:24kB active_file:232kB inactive_file:368kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4332kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957144kB kernel_stack:16656kB pagetables:1184kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1121 all_unreclaimable? yes [165793.192765] lowmem_reserve[]: 0 0 0 0 [165793.206075] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165793.252645] Node 0 DMA32: 45*4kB (UEM) 1860*8kB (UEM) 1334*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59476kB [165793.303574] Node 0 Normal: 158*4kB (UEM) 147*8kB (UEM) 118*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36944kB [165793.356312] Node 1 Normal: 571*4kB (UEM) 421*8kB (UEM) 240*16kB (UEM) 351*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45620kB [165793.407981] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165793.438027] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165793.467215] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165793.497255] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165793.526423] 1786 total pagecache pages [165793.540136] 0 pages in swap cache [165793.552418] Swap cache stats: add 314837, delete 314837, find 93770/151419 [165793.576440] Free swap = 16277404kB [165793.589307] Total swap = 16319484kB [165793.602195] 8369063 pages RAM [165793.613339] 0 pages HighMem/MovableOnly [165793.627342] 241261 pages reserved [165793.639627] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165793.666812] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165793.696572] [ 1046] 0 1046 1588718 3709 132 0 -1000 multipathd [165793.725479] [ 2373] 0 2373 13863 113 27 120 -1000 auditd [165793.753241] [ 2444] 81 2444 9196 200 21 149 -900 dbus-daemon [165793.782444] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165793.809656] [67635] 0 67635 90 11 4 15 0 systemd-cgroups [165793.839949] [67641] 0 67641 79 8 4 7 0 systemd-cgroups [165793.870230] [67646] 0 67646 610 26 4 13 0 systemd-cgroups [165793.900467] [67647] 0 67647 79 0 4 6 0 systemd-cgroups [165793.930688] [67649] 0 67649 79 0 4 11 0 systemd-cgroups [165793.960890] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165793.991054] [67651] 0 67651 79 0 3 6 0 systemd-cgroups [165794.021190] [67652] 0 67652 610 26 5 13 0 systemd-cgroups [165794.051307] Out of memory: Kill process 67652 (systemd-cgroups) score 0 or sacrifice child [165794.079695] Killed process 67652 (systemd-cgroups) total-vm:2440kB, anon-rss:0kB, file-rss:104kB, shmem-rss:0kB [165794.119698] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165794.148210] systemd-cgroups cpuset=/ mems_allowed=0-1 [165794.166232] CPU: 23 PID: 67641 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165794.206334] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165794.244439] ffff88033376bf40 00000000c27ed5d1 ffff8803e416b988 ffffffff816a3e2d [165794.269999] ffff8803e416ba18 ffffffff8169f246 ffffffff810e93ac ffff8807f5633910 [165794.295535] ffff8807f5633928 0000000000000206 0000000000000206 ffff8803e416ba08 [165794.321058] Call Trace: [165794.330206] [] dump_stack+0x19/0x1b [165794.348207] [] dump_header+0x90/0x229 [165794.366775] [] ? ktime_get_ts64+0x4c/0xf0 [165794.386471] [] ? delayacct_end+0x8f/0xb0 [165794.405862] [] oom_kill_process+0x254/0x3d0 [165794.426087] [] ? oom_unkillable_task+0xcd/0x120 [165794.447435] [] ? find_lock_task_mm+0x56/0xc0 [165794.467915] [] out_of_memory+0x4b6/0x4f0 [165794.487214] [] __alloc_pages_slowpath+0x5d6/0x724 [165794.509073] [] __alloc_pages_nodemask+0x405/0x420 [165794.530906] [] alloc_pages_vma+0xb5/0x200 [165794.550430] [] read_swap_cache_async+0xed/0x160 [165794.571659] [] swapin_readahead+0xa8/0x110 [165794.591484] [] handle_mm_fault+0xadb/0xfa0 [165794.611322] [] __do_page_fault+0x154/0x450 [165794.631151] [] do_page_fault+0x35/0x90 [165794.649835] [] page_fault+0x28/0x30 [165794.667625] Mem-Info: [165794.675990] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:243 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560188 mapped:1547 shmem:0 pagetables:322 bounce:0 free:39405 free_pcp:30 free_cma:0 [165794.787377] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165794.926616] lowmem_reserve[]: 0 2580 15620 15620 [165794.942874] Node 0 DMA32 free:59476kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1616kB pagetables:20kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5336 all_unreclaimable? yes [165795.092857] lowmem_reserve[]: 0 0 13040 13040 [165795.108370] Node 0 Normal free:36944kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:588kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14080kB pagetables:84kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:899 all_unreclaimable? yes [165795.262630] lowmem_reserve[]: 0 0 0 0 [165795.275921] Node 1 Normal free:45628kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:332kB inactive_file:268kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4504kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957144kB kernel_stack:16656kB pagetables:1184kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1224 all_unreclaimable? yes [165795.432571] lowmem_reserve[]: 0 0 0 0 [165795.445883] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165795.492414] Node 0 DMA32: 44*4kB (UEM) 1860*8kB (UEM) 1335*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59488kB [165795.543328] Node 0 Normal: 168*4kB (UEM) 147*8kB (UEM) 118*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36984kB [165795.596065] Node 1 Normal: 572*4kB (UEM) 421*8kB (UEM) 239*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45640kB [165795.647723] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165795.677765] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165795.706942] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165795.736968] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165795.766129] 1780 total pagecache pages [165795.779841] 1 pages in swap cache [165795.792120] Swap cache stats: add 314840, delete 314839, find 93770/151419 [165795.816142] Free swap = 16277448kB [165795.829012] Total swap = 16319484kB [165795.841877] 8369063 pages RAM [165795.853021] 0 pages HighMem/MovableOnly [165795.867022] 241261 pages reserved [165795.879294] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165795.906453] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165795.936202] [ 1046] 0 1046 1588718 3694 132 0 -1000 multipathd [165795.965072] [ 2373] 0 2373 13863 113 27 120 -1000 auditd [165795.992872] [ 2444] 81 2444 9196 200 21 149 -900 dbus-daemon [165796.022022] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165796.049217] [67635] 0 67635 90 11 4 15 0 systemd-cgroups [165796.079452] [67641] 0 67641 79 8 4 7 0 systemd-cgroups [165796.109703] [67646] 0 67646 610 26 4 13 0 systemd-cgroups [165796.139948] [67647] 0 67647 79 0 4 6 0 systemd-cgroups [165796.170158] [67649] 0 67649 79 0 4 11 0 systemd-cgroups [165796.200344] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165796.230505] [67651] 0 67651 79 0 3 6 0 systemd-cgroups [165796.260622] Out of memory: Kill process 67646 (systemd-cgroups) score 0 or sacrifice child [165796.289054] Killed process 67646 (systemd-cgroups) total-vm:2440kB, anon-rss:0kB, file-rss:104kB, shmem-rss:0kB [165796.324379] systemd-cgroups: page allocation failure: order:0, mode:0x200da [165796.349141] CPU: 15 PID: 67646 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165796.389880] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165796.428586] 00000000000200da 000000004f3cc0cd ffff880083553a98 ffffffff816a3e2d [165796.454744] ffff880083553b28 ffffffff81188820 ffffffff8118b540 0000000000000000 [165796.480912] ffffffffffffffff 000200da00000000 ffff880083553b28 000000004f3cc0cd [165796.507053] Call Trace: [165796.516783] [] dump_stack+0x19/0x1b [165796.535370] [] warn_alloc_failed+0x110/0x180 [165796.556069] [] ? drain_pages+0xb0/0xb0 [165796.575007] [] ? wake_up_atomic_t+0x30/0x30 [165796.595373] [] __alloc_pages_slowpath+0x6b6/0x724 [165796.617437] [] __alloc_pages_nodemask+0x405/0x420 [165796.639471] [] alloc_pages_vma+0xb5/0x200 [165796.659412] [] read_swap_cache_async+0xed/0x160 [165796.681042] [] swapin_readahead+0xa8/0x110 [165796.701175] [] handle_mm_fault+0xadb/0xfa0 [165796.721279] [] __do_page_fault+0x154/0x450 [165796.741374] [] do_page_fault+0x35/0x90 [165796.760288] [] page_fault+0x28/0x30 [165796.778349] Mem-Info: [165796.786952] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:140 inactive_file:93 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560188 mapped:1547 shmem:0 pagetables:317 bounce:0 free:39425 free_pcp:0 free_cma:0 [165796.899832] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165797.039465] lowmem_reserve[]: 0 2580 15620 15620 [165797.055821] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:72kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1600kB pagetables:20kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:37408 all_unreclaimable? no [165797.206548] lowmem_reserve[]: 0 0 13040 13040 [165797.222337] Node 0 Normal free:37000kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:256kB inactive_file:52kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14064kB pagetables:68kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:462 all_unreclaimable? yes [165797.376887] lowmem_reserve[]: 0 0 0 0 [165797.390239] Node 1 Normal free:45640kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957144kB kernel_stack:16640kB pagetables:1180kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1150 all_unreclaimable? yes [165797.546737] lowmem_reserve[]: 0 0 0 0 [165797.560115] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165797.608019] Node 0 DMA32: 44*4kB (UEM) 1860*8kB (UEM) 1335*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59488kB [165797.660268] Node 0 Normal: 168*4kB (UEM) 147*8kB (UEM) 119*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37000kB [165797.713226] Node 1 Normal: 572*4kB (UEM) 421*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45688kB [165797.765127] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165797.795277] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165797.824559] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165797.854678] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165797.884301] 1780 total pagecache pages [165797.898306] 1 pages in swap cache [165797.911017] Swap cache stats: add 314848, delete 314847, find 93773/151424 [165797.935493] Free swap = 16277448kB [165797.948837] Total swap = 16319484kB [165797.962121] 8369063 pages RAM [165797.973710] 0 pages HighMem/MovableOnly [165797.988189] 241261 pages reserved [165798.001199] ll_ost_io01_104 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [165798.029993] ll_ost_io01_104 cpuset=/ mems_allowed=1 [165798.047926] CPU: 9 PID: 226541 Comm: ll_ost_io01_104 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165798.089222] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165798.128036] ffff88081d3a8fd0 00000000c8832be1 ffff8807947af998 ffffffff816a3e2d [165798.154238] ffff8807947afa28 ffffffff8169f246 ffffffff810e93ac ffff8807ebda5de0 [165798.180563] ffff8807ebda5df8 0000000000000206 0000000000000206 ffff8807947afa18 [165798.206709] Call Trace: [165798.216402] [] dump_stack+0x19/0x1b [165798.234983] [] dump_header+0x90/0x229 [165798.254281] [] ? ktime_get_ts64+0x4c/0xf0 [165798.274518] [] ? delayacct_end+0x8f/0xb0 [165798.294447] [] oom_kill_process+0x254/0x3d0 [165798.315332] [] ? oom_unkillable_task+0xcd/0x120 [165798.337108] [] ? find_lock_task_mm+0x56/0xc0 [165798.358003] [] out_of_memory+0x4b6/0x4f0 [165798.377789] [] __alloc_pages_slowpath+0x5d6/0x724 [165798.400305] [] __alloc_pages_nodemask+0x405/0x420 [165798.422670] [] __vmalloc_node_range+0x12f/0x270 [165798.444535] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165798.468423] [] vzalloc_node+0x47/0x50 [165798.487335] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165798.511517] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165798.534780] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [165798.558575] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [165798.580670] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [165798.606459] [] kthread+0xcf/0xe0 [165798.623590] [] ? insert_kthread_work+0x40/0x40 [165798.645033] [] ret_from_fork+0x58/0x90 [165798.664145] [] ? insert_kthread_work+0x40/0x40 [165798.685482] Mem-Info: [165798.694081] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:206 inactive_file:24 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560176 mapped:1547 shmem:0 pagetables:313 bounce:0 free:39344 free_pcp:0 free_cma:0 [165798.807042] Node 1 Normal free:45440kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957096kB kernel_stack:16640kB pagetables:1164kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1238 all_unreclaimable? yes [165798.963235] lowmem_reserve[]: 0 0 0 0 [165798.976693] Node 1 Normal: 580*4kB (UEM) 421*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45720kB [165799.028481] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165799.058584] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165799.087795] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165799.117559] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165799.146710] 1780 total pagecache pages [165799.160293] 0 pages in swap cache [165799.172339] Swap cache stats: add 314851, delete 314851, find 93773/151424 [165799.196206] Free swap = 16277492kB [165799.208743] Total swap = 16319484kB [165799.221510] 8369063 pages RAM [165799.232535] 0 pages HighMem/MovableOnly [165799.246226] 241261 pages reserved [165799.258213] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165799.285299] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165799.314925] [ 1046] 0 1046 1588718 3738 132 0 -1000 multipathd [165799.343745] [ 2373] 0 2373 13863 113 27 121 -1000 auditd [165799.371379] [ 2444] 81 2444 9196 200 21 149 -900 dbus-daemon [165799.400421] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165799.427347] [67635] 0 67635 90 11 4 15 0 systemd-cgroups [165799.457556] [67641] 0 67641 79 8 4 7 0 systemd-cgroups [165799.487769] [67647] 0 67647 79 0 4 6 0 systemd-cgroups [165799.517972] [67649] 0 67649 79 0 4 13 0 systemd-cgroups [165799.548166] [67650] 0 67650 79 0 4 4 0 systemd-cgroups [165799.578320] [67651] 0 67651 79 0 3 6 0 systemd-cgroups [165799.608500] Out of memory: Kill process 67635 (systemd-cgroups) score 0 or sacrifice child [165799.636961] Killed process 67635 (systemd-cgroups) total-vm:360kB, anon-rss:0kB, file-rss:44kB, shmem-rss:0kB [165799.673860] systemd-cgroups: page allocation failure: order:0, mode:0x201da [165799.697885] CPU: 16 PID: 67635 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165799.737804] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165799.776232] 00000000000201da 000000008d373a7b ffff8802763b3a30 ffffffff816a3e2d [165799.802151] ffff8802763b3ac0 ffffffff81188820 ffffffff8118b540 0000000000000000 [165799.828124] ffffffffffffffff 000201da00000000 ffff8802763b3ac0 000000008d373a7b [165799.854071] Call Trace: [165799.863648] [] dump_stack+0x19/0x1b [165799.881650] [] warn_alloc_failed+0x110/0x180 [165799.902225] [] ? drain_pages+0xb0/0xb0 [165799.921082] [] ? wake_up_atomic_t+0x30/0x30 [165799.941703] [] __alloc_pages_slowpath+0x6b6/0x724 [165799.964049] [] __alloc_pages_nodemask+0x405/0x420 [165799.986348] [] alloc_pages_current+0x98/0x110 [165800.007485] [] __page_cache_alloc+0x97/0xb0 [165800.028018] [] filemap_fault+0x170/0x410 [165800.047678] [] ext4_filemap_fault+0x36/0x50 [ext4] [165800.070167] [] __do_fault+0x52/0xe0 [165800.088357] [] do_read_fault.isra.44+0x4b/0x130 [165800.109964] [] handle_mm_fault+0x691/0xfa0 [165800.130119] [] __do_page_fault+0x154/0x450 [165800.150237] [] do_page_fault+0x35/0x90 [165800.169209] [] page_fault+0x28/0x30 [165800.187333] Mem-Info: [165800.196019] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:179 inactive_file:49 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560176 mapped:1547 shmem:0 pagetables:313 bounce:0 free:39444 free_pcp:0 free_cma:0 [165800.309274] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165800.449930] lowmem_reserve[]: 0 2580 15620 15620 [165800.466524] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:8kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1255 all_unreclaimable? no [165800.617367] lowmem_reserve[]: 0 0 13040 13040 [165800.633250] Node 0 Normal free:36996kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:76kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14064kB pagetables:68kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:982 all_unreclaimable? yes [165800.789843] lowmem_reserve[]: 0 0 0 0 [165800.803599] Node 1 Normal free:45716kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957096kB kernel_stack:16640kB pagetables:1164kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1039 all_unreclaimable? yes [165800.961859] lowmem_reserve[]: 0 0 0 0 [165800.975582] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165801.023082] Node 0 DMA32: 42*4kB (UEM) 1860*8kB (UEM) 1336*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165801.074934] Node 0 Normal: 167*4kB (UEM) 147*8kB (UEM) 119*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36996kB [165801.128646] Node 1 Normal: 579*4kB (UEM) 421*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [165801.181342] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165801.211877] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165801.241550] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165801.271597] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165801.300766] 1780 total pagecache pages [165801.314494] 1 pages in swap cache [165801.326783] Swap cache stats: add 314854, delete 314853, find 93774/151426 [165801.350807] Free swap = 16277492kB [165801.363685] Total swap = 16319484kB [165801.376561] 8369063 pages RAM [165801.387713] 0 pages HighMem/MovableOnly [165801.401725] 241261 pages reserved [165801.414273] kworker/u481:2 invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [165801.443078] kworker/u481:2 cpuset=/ mems_allowed=0-1 [165801.461124] CPU: 23 PID: 67657 Comm: kworker/u481:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165801.501526] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165801.539938] ffff88042d0c6eb0 00000000c739d5f5 ffff88082e637890 ffffffff816a3e2d [165801.565813] ffff88082e637920 ffffffff8169f246 ffffffff810e93ac ffff8807ebebe750 [165801.591661] ffff8807ebebe768 0000000000000202 0000000000000202 ffff88082e637910 [165801.617493] Call Trace: [165801.626939] [] dump_stack+0x19/0x1b [165801.645260] [] dump_header+0x90/0x229 [165801.664128] [] ? ktime_get_ts64+0x4c/0xf0 [165801.684113] [] ? delayacct_end+0x8f/0xb0 [165801.703785] [] oom_kill_process+0x254/0x3d0 [165801.724292] [] ? oom_unkillable_task+0xcd/0x120 [165801.745927] [] ? find_lock_task_mm+0x56/0xc0 [165801.766686] [] out_of_memory+0x4b6/0x4f0 [165801.786290] [] __alloc_pages_slowpath+0x5d6/0x724 [165801.808433] [] __alloc_pages_nodemask+0x405/0x420 [165801.830557] [] alloc_pages_vma+0xb5/0x200 [165801.850368] [] ? anon_vma_interval_tree_insert+0x8b/0x90 [165801.874472] [] handle_mm_fault+0xb60/0xfa0 [165801.894553] [] ? follow_page_mask+0x37/0x780 [165801.915186] [] __get_user_pages+0x17e/0x600 [165801.935513] [] get_user_pages+0x52/0x60 [165801.954674] [] copy_strings.isra.20+0x265/0x340 [165801.976108] [] copy_strings_kernel+0x34/0x40 [165801.996679] [] do_execve_common.isra.25+0x546/0x6c0 [165802.019235] [] ? vfs_rename+0x860/0x930 [165802.038333] [] do_execve+0x18/0x20 [165802.055981] [] ____call_usermodehelper+0xfc/0x130 [165802.077903] [] ? ____call_usermodehelper+0x130/0x130 [165802.100662] [] call_helper+0x1e/0x20 [165802.118822] [] ret_from_fork+0x58/0x90 [165802.137533] [] ? ____call_usermodehelper+0x130/0x130 [165802.160225] Mem-Info: [165802.168599] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:122 inactive_file:112 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560176 mapped:1547 shmem:0 pagetables:314 bounce:0 free:39443 free_pcp:0 free_cma:0 [165802.280425] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165802.419775] lowmem_reserve[]: 0 2580 15620 15620 [165802.436067] Node 0 DMA32 free:59112kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:0kB inactive_file:48kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:364kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4967 all_unreclaimable? yes [165802.586228] lowmem_reserve[]: 0 0 13040 13040 [165802.601658] Node 0 Normal free:36900kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:76kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14064kB pagetables:56kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:658 all_unreclaimable? yes [165802.755621] lowmem_reserve[]: 0 0 0 0 [165802.768824] Node 1 Normal free:45728kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957096kB kernel_stack:16640kB pagetables:1160kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:930 all_unreclaimable? yes [165802.924474] lowmem_reserve[]: 0 0 0 0 [165802.937783] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165802.984349] Node 0 DMA32: 40*4kB (UEM) 1860*8kB (UEM) 1336*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59488kB [165803.035287] Node 0 Normal: 169*4kB (UEM) 149*8kB (UEM) 119*16kB (UEM) 229*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37020kB [165803.088012] Node 1 Normal: 580*4kB (UEM) 422*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45728kB [165803.139659] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165803.169687] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165803.198860] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165803.228873] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165803.258044] 1780 total pagecache pages [165803.271798] 10 pages in swap cache [165803.284399] Swap cache stats: add 314880, delete 314872, find 93789/151450 [165803.308402] Free swap = 16277520kB [165803.321274] Total swap = 16319484kB [165803.334168] 8369063 pages RAM [165803.345288] 0 pages HighMem/MovableOnly [165803.359293] 241261 pages reserved [165803.371602] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165803.398808] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165803.428575] [ 1046] 0 1046 1588718 3718 132 0 -1000 multipathd [165803.457500] [ 2373] 0 2373 13863 112 27 122 -1000 auditd [165803.485268] [ 2444] 81 2444 9196 199 21 149 -900 dbus-daemon [165803.514437] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165803.541668] [67641] 0 67641 79 8 4 8 0 systemd-cgroups [165803.571968] [67647] 0 67647 79 0 4 10 0 systemd-cgroups [165803.602227] [67649] 0 67649 79 0 4 16 0 systemd-cgroups [165803.610120] Lustre: soaked-OST0016: Client 1dc01503-bd29-4e1d-18f1-96cbe7715c08 (at 192.168.1.120@o2ib) reconnecting [165803.610122] Lustre: Skipped 34 previous similar messages [165803.687253] [67650] 0 67650 79 0 4 9 0 systemd-cgroups [165803.717487] [67651] 0 67651 79 0 3 6 0 systemd-cgroups [165803.747718] Out of memory: Kill process 67641 (systemd-cgroups) score 0 or sacrifice child [165803.776196] Killed process 67641 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:32kB, shmem-rss:0kB [165803.810732] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165803.839810] systemd-cgroups cpuset=/ mems_allowed=0-1 [165803.858114] CPU: 8 PID: 67649 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165803.898763] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165803.937354] ffff8805acfabf40 00000000ac06895f ffff88042d8a7988 ffffffff816a3e2d [165803.963302] ffff88042d8a7a18 ffffffff8169f246 ffffffff810e93ac ffff8802b1253b40 [165803.989159] ffff8802b1253b58 0000000000000206 0000000000000206 ffff88042d8a7a08 [165804.015167] Call Trace: [165804.024787] [] dump_stack+0x19/0x1b [165804.043069] [] dump_header+0x90/0x229 [165804.061796] [] ? ktime_get_ts64+0x4c/0xf0 [165804.081659] [] ? delayacct_end+0x8f/0xb0 [165804.101221] [] oom_kill_process+0x254/0x3d0 [165804.121703] [] ? oom_unkillable_task+0xcd/0x120 [165804.143141] [] ? find_lock_task_mm+0x56/0xc0 [165804.163836] [] out_of_memory+0x4b6/0x4f0 [165804.183327] [] __alloc_pages_slowpath+0x5d6/0x724 [165804.205337] [] __alloc_pages_nodemask+0x405/0x420 [165804.227323] [] alloc_pages_vma+0xb5/0x200 [165804.247034] [] read_swap_cache_async+0xed/0x160 [165804.268406] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [165804.291262] [] swapin_readahead+0xa8/0x110 [165804.311256] [] handle_mm_fault+0xadb/0xfa0 [165804.331245] [] __do_page_fault+0x154/0x450 [165804.351261] [] do_page_fault+0x35/0x90 [165804.370121] [] page_fault+0x28/0x30 [165804.388094] Mem-Info: [165804.396572] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:87 inactive_file:147 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560176 mapped:1549 shmem:0 pagetables:306 bounce:0 free:39192 free_pcp:163 free_cma:0 [165804.509483] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165804.649374] lowmem_reserve[]: 0 2580 15620 15620 [165804.665820] Node 0 DMA32 free:59116kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:20kB unstable:0kB bounce:0kB free_pcp:372kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:8920 all_unreclaimable? yes [165804.817895] lowmem_reserve[]: 0 0 13040 13040 [165804.833558] Node 0 Normal free:36344kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:12kB active_file:100kB inactive_file:208kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1540kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14064kB pagetables:48kB unstable:0kB bounce:0kB free_pcp:284kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3753 all_unreclaimable? yes [165804.990058] lowmem_reserve[]: 0 0 0 0 [165805.003469] Node 1 Normal free:45728kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957096kB kernel_stack:16640kB pagetables:1156kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1043 all_unreclaimable? yes [165805.160069] lowmem_reserve[]: 0 0 0 0 [165805.173527] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165805.220266] Node 0 DMA32: 39*4kB (UEM) 1858*8kB (UE) 1336*16kB (UE) 498*32kB (UEM) 100*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59116kB [165805.270970] Node 0 Normal: 142*4kB (UEM) 145*8kB (UEM) 114*16kB (UEM) 225*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36672kB [165805.323735] Node 1 Normal: 582*4kB (UEM) 422*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45736kB [165805.376044] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165805.406213] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165805.435538] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165805.465784] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165805.495052] 1790 total pagecache pages [165805.508932] 1 pages in swap cache [165805.521326] Swap cache stats: add 314891, delete 314890, find 93792/151456 [165805.545513] Free swap = 16277536kB [165805.558562] Total swap = 16319484kB [165805.571622] 8369063 pages RAM [165805.582943] 0 pages HighMem/MovableOnly [165805.597083] 241261 pages reserved [165805.609532] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165805.636821] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165805.666787] [ 1046] 0 1046 1588718 3717 132 0 -1000 multipathd [165805.695842] [ 2373] 0 2373 13863 112 27 124 -1000 auditd [165805.723803] [ 2444] 81 2444 9196 199 21 150 -900 dbus-daemon [165805.753112] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165805.780424] [67647] 0 67647 79 0 4 10 0 systemd-cgroups [165805.810837] [67649] 0 67649 79 0 4 16 0 systemd-cgroups [165805.841186] [67650] 0 67650 79 0 4 11 0 systemd-cgroups [165805.871585] [67651] 0 67651 79 0 3 7 0 systemd-cgroups [165805.901872] [67657] 0 67657 41 0 2 1 0 systemd-cgroups [165805.932179] Out of memory: Kill process 67649 (systemd-cgroups) score 0 or sacrifice child [165805.960717] Killed process 67649 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [165805.994960] ll_ost_out01_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [165806.023713] ll_ost_out01_00 cpuset=/ mems_allowed=1 [165806.041382] CPU: 14 PID: 183717 Comm: ll_ost_out01_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165806.082676] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165806.120861] ffff8803dd56eeb0 0000000085bdad45 ffff8803966af998 ffffffff816a3e2d [165806.146483] ffff8803966afa28 ffffffff8169f246 ffffffff810e93ac ffff8807f1e228a0 [165806.172091] ffff8807f1e228b8 0000000000000206 0000000000000206 ffff8803966afa18 [165806.197777] Call Trace: [165806.207081] [] dump_stack+0x19/0x1b [165806.225241] [] dump_header+0x90/0x229 [165806.243956] [] ? ktime_get_ts64+0x4c/0xf0 [165806.264039] [] ? delayacct_end+0x8f/0xb0 [165806.283803] [] oom_kill_process+0x254/0x3d0 [165806.304421] [] ? oom_unkillable_task+0xcd/0x120 [165806.326167] [] ? find_lock_task_mm+0x56/0xc0 [165806.347022] [] out_of_memory+0x4b6/0x4f0 [165806.366826] [] __alloc_pages_slowpath+0x5d6/0x724 [165806.388768] [] __alloc_pages_nodemask+0x405/0x420 [165806.410685] [] __vmalloc_node_range+0x12f/0x270 [165806.432052] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165806.455660] [] vzalloc_node+0x47/0x50 [165806.474109] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165806.497699] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [165806.520725] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [165806.544330] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [165806.565939] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [165806.591241] [] kthread+0xcf/0xe0 [165806.608244] [] ? insert_kthread_work+0x40/0x40 [165806.629248] [] ret_from_fork+0x58/0x90 [165806.647969] [] ? insert_kthread_work+0x40/0x40 [165806.668971] Mem-Info: [165806.677390] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:77 inactive_file:160 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560178 mapped:1549 shmem:0 pagetables:303 bounce:0 free:39298 free_pcp:0 free_cma:0 [165806.789066] Node 1 Normal free:45432kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:240kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4320kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957104kB kernel_stack:16640kB pagetables:1156kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1210 all_unreclaimable? yes [165806.944218] lowmem_reserve[]: 0 0 0 0 [165806.957398] Node 1 Normal: 563*4kB (UEM) 396*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45452kB [165807.008541] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165807.038291] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165807.067160] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165807.096880] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165807.125712] 1785 total pagecache pages [165807.139082] 1 pages in swap cache [165807.150992] Swap cache stats: add 314897, delete 314896, find 93792/151457 [165807.174626] Free swap = 16277560kB [165807.187073] Total swap = 16319484kB [165807.199539] 8369063 pages RAM [165807.210273] 0 pages HighMem/MovableOnly [165807.223867] 241261 pages reserved [165807.235747] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165807.262533] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165807.291894] [ 1046] 0 1046 1588718 3716 132 0 -1000 multipathd [165807.320407] [ 2373] 0 2373 13863 110 27 124 -1000 auditd [165807.347762] [ 2444] 81 2444 9196 198 21 150 -900 dbus-daemon [165807.376560] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165807.403387] [67647] 0 67647 79 0 4 10 0 systemd-cgroups [165807.433324] [67650] 0 67650 79 0 4 11 0 systemd-cgroups [165807.463293] [67651] 0 67651 79 0 3 7 0 systemd-cgroups [165807.493218] [67657] 0 67657 41 0 2 1 0 systemd-cgroups [165807.523135] Out of memory: Kill process 67650 (systemd-cgroups) score 0 or sacrifice child [165807.551340] Killed process 67650 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [165807.585831] systemd-cgroups: page allocation failure: order:0, mode:0x200da [165807.610046] CPU: 25 PID: 67650 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165807.649951] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165807.688043] 00000000000200da 000000006b8188ac ffff88042d89fa90 ffffffff816a3e2d [165807.713640] ffff88042d89fb20 ffffffff81188820 ffffffff8118b540 0000000000000000 [165807.739241] ffffffffffffffff 000200da00000000 ffff88042d89fb20 000000006b8188ac [165807.764742] Call Trace: [165807.773964] [] dump_stack+0x19/0x1b [165807.792048] [] warn_alloc_failed+0x110/0x180 [165807.812705] [] ? drain_pages+0xb0/0xb0 [165807.831863] [] ? wake_up_atomic_t+0x30/0x30 [165807.852458] [] __alloc_pages_slowpath+0x6b6/0x724 [165807.874471] [] __alloc_pages_nodemask+0x405/0x420 [165807.896477] [] alloc_pages_vma+0xb5/0x200 [165807.916192] [] wp_page_copy.isra.58+0x88/0x470 [165807.937330] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [165807.960406] [] do_wp_page+0xeb/0x5c0 [165807.978635] [] ? do_read_fault.isra.44+0xe6/0x130 [165808.000560] [] handle_mm_fault+0x70c/0xfa0 [165808.020464] [] __do_page_fault+0x154/0x450 [165808.040644] [] do_page_fault+0x35/0x90 [165808.059647] [] page_fault+0x28/0x30 [165808.077761] Mem-Info: [165808.086418] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:104 inactive_file:130 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560178 mapped:1547 shmem:0 pagetables:303 bounce:0 free:39307 free_pcp:0 free_cma:0 [165808.200881] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165808.341220] lowmem_reserve[]: 0 2580 15620 15620 [165808.357771] Node 0 DMA32 free:59360kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:64kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:16kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:13144 all_unreclaimable? yes [165808.510879] lowmem_reserve[]: 0 0 13040 13040 [165808.526758] Node 0 Normal free:36752kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:140kB inactive_file:172kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14016kB pagetables:40kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:468 all_unreclaimable? yes [165808.683716] lowmem_reserve[]: 0 0 0 0 [165808.697476] Node 1 Normal free:45452kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4324kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957104kB kernel_stack:16640kB pagetables:1156kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1018 all_unreclaimable? yes [165808.855842] lowmem_reserve[]: 0 0 0 0 [165808.869595] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165808.917087] Node 0 DMA32: 39*4kB (UEM) 1860*8kB (UEM) 1336*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59484kB [165808.968983] Node 0 Normal: 148*4kB (UEM) 146*8kB (UEM) 115*16kB (UEM) 226*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36752kB [165809.022728] Node 1 Normal: 563*4kB (UEM) 396*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45452kB [165809.075457] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165809.106008] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165809.135696] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165809.166213] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165809.195869] 1781 total pagecache pages [165809.210076] 0 pages in swap cache [165809.222850] Swap cache stats: add 314898, delete 314898, find 93792/151457 [165809.247358] Free swap = 16277560kB [165809.260719] Total swap = 16319484kB [165809.274075] 8369063 pages RAM [165809.285693] 0 pages HighMem/MovableOnly [165809.300177] 241261 pages reserved [165809.315570] auditd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=-1000 [165809.342783] auditd cpuset=/ mems_allowed=0-1 [165809.358247] CPU: 1 PID: 2373 Comm: auditd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165809.394614] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165809.432997] ffff88082c1caf70 00000000689d2a54 ffff88082ae7f988 ffffffff816a3e2d [165809.458838] ffff88082ae7fa18 ffffffff8169f246 ffffffff810e93ac ffff8808162f74b0 [165809.484642] ffff8808162f74c8 0000000000000206 0000000000000206 ffff88082ae7fa08 [165809.510440] Call Trace: [165809.519873] [] dump_stack+0x19/0x1b [165809.538155] [] dump_header+0x90/0x229 [165809.556993] [] ? ktime_get_ts64+0x4c/0xf0 [165809.576954] [] ? delayacct_end+0x8f/0xb0 [165809.596600] [] oom_kill_process+0x254/0x3d0 [165809.617084] [] ? oom_unkillable_task+0xcd/0x120 [165809.638690] [] ? find_lock_task_mm+0x56/0xc0 [165809.659434] [] out_of_memory+0x4b6/0x4f0 [165809.679013] [] __alloc_pages_slowpath+0x5d6/0x724 [165809.701152] [] __alloc_pages_nodemask+0x405/0x420 [165809.723268] [] alloc_pages_vma+0xb5/0x200 [165809.743073] [] read_swap_cache_async+0xed/0x160 [165809.764576] [] ? mem_cgroup_update_page_stat+0x16/0x50 [165809.788084] [] swapin_readahead+0xa8/0x110 [165809.808157] [] handle_mm_fault+0xadb/0xfa0 [165809.828202] [] __do_page_fault+0x154/0x450 [165809.848210] [] do_page_fault+0x35/0x90 [165809.867095] [] page_fault+0x28/0x30 [165809.885054] Mem-Info: [165809.893570] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:141 inactive_file:88 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560178 mapped:1545 shmem:0 pagetables:303 bounce:0 free:39316 free_pcp:0 free_cma:0 [165810.005455] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165810.144728] lowmem_reserve[]: 0 2580 15620 15620 [165810.160971] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:44kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:16kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6526 all_unreclaimable? yes [165810.310608] lowmem_reserve[]: 0 0 13040 13040 [165810.326085] Node 0 Normal free:36752kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:288kB inactive_file:24kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14016kB pagetables:40kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1465 all_unreclaimable? yes [165810.480712] lowmem_reserve[]: 0 0 0 0 [165810.493988] Node 1 Normal free:45452kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4308kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957104kB kernel_stack:16640kB pagetables:1156kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1275 all_unreclaimable? yes [165810.650044] lowmem_reserve[]: 0 0 0 0 [165810.663354] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165810.709928] Node 0 DMA32: 41*4kB (UEM) 1860*8kB (UEM) 1336*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59492kB [165810.760855] Node 0 Normal: 156*4kB (UEM) 146*8kB (UEM) 115*16kB (UEM) 226*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36784kB [165810.813606] Node 1 Normal: 563*4kB (UEM) 396*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45452kB [165810.865291] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165810.895335] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165810.924518] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165810.954550] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165810.983717] 1781 total pagecache pages [165810.997433] 0 pages in swap cache [165811.009717] Swap cache stats: add 314899, delete 314899, find 93792/151457 [165811.033744] Free swap = 16277580kB [165811.046615] Total swap = 16319484kB [165811.059497] 8369063 pages RAM [165811.070642] 0 pages HighMem/MovableOnly [165811.084648] 241261 pages reserved [165811.096928] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165811.124122] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165811.153881] [ 1046] 0 1046 1588718 3709 132 0 -1000 multipathd [165811.182786] [ 2373] 0 2373 13863 110 27 124 -1000 auditd [165811.210544] [ 2444] 81 2444 9196 198 21 151 -900 dbus-daemon [165811.239732] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165811.266937] [67647] 0 67647 79 0 4 10 0 systemd-cgroups [165811.297234] [67651] 0 67651 79 0 3 7 0 systemd-cgroups [165811.327499] [67657] 0 67657 41 0 2 1 0 systemd-cgroups [165811.357735] Out of memory: Kill process 67647 (systemd-cgroups) score 0 or sacrifice child [165811.386253] Killed process 67647 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [165811.423051] kworker/u480:2 invoked oom-killer: gfp_mask=0x202d2, order=3, oom_score_adj=0 [165811.451337] kworker/u480:2 cpuset=/ mems_allowed=0-1 [165811.468956] CPU: 21 PID: 67636 Comm: kworker/u480:2 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165811.508960] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165811.547521] Workqueue: rdma_cm cma_work_handler [rdma_cm] [165811.566580] ffff88082cd58fd0 00000000987a9902 ffff88052cdf7668 ffffffff816a3e2d [165811.592244] ffff88052cdf76f8 ffffffff8169f246 ffffffff810e93ac ffff8807ebebed00 [165811.617892] ffff8807ebebed18 0000000000000206 0000000000000206 ffff88052cdf76e8 [165811.643522] Call Trace: [165811.652791] [] dump_stack+0x19/0x1b [165811.670936] [] dump_header+0x90/0x229 [165811.689610] [] ? ktime_get_ts64+0x4c/0xf0 [165811.709412] [] ? delayacct_end+0x8f/0xb0 [165811.728903] [] oom_kill_process+0x254/0x3d0 [165811.749227] [] ? oom_unkillable_task+0xcd/0x120 [165811.770694] [] ? find_lock_task_mm+0x56/0xc0 [165811.791285] [] out_of_memory+0x4b6/0x4f0 [165811.810713] [] __alloc_pages_slowpath+0x5d6/0x724 [165811.832695] [] __alloc_pages_nodemask+0x405/0x420 [165811.854670] [] mlx4_alloc_icm+0x3a2/0x660 [mlx4_core] [165811.877764] [] mlx4_table_get+0xa9/0x120 [mlx4_core] [165811.900550] [] __mlx4_qp_alloc_icm+0x31/0x130 [mlx4_core] [165811.924746] [] mlx4_qp_alloc+0x42/0x130 [mlx4_core] [165811.947211] [] create_qp_common+0x978/0x1000 [mlx4_ib] [165811.970515] [] ? dma_pool_free+0xa7/0xd0 [165811.989831] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [165812.013160] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [165812.037071] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [165812.060972] [] ib_create_qp+0x7a/0x2f0 [ib_core] [165812.082583] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [165812.104477] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [165812.128661] [] kiblnd_cm_callback+0x145f/0x2380 [ko2iblnd] [165812.153130] [] cma_work_handler+0x6c/0xa0 [rdma_cm] [165812.175602] [] process_one_work+0x17a/0x440 [165812.195787] [] worker_thread+0x126/0x3c0 [165812.215115] [] ? manage_workers.isra.24+0x2a0/0x2a0 [165812.237588] [] kthread+0xcf/0xe0 [165812.254614] [] ? insert_kthread_work+0x40/0x40 [165812.275651] [] ret_from_fork+0x58/0x90 [165812.294395] [] ? insert_kthread_work+0x40/0x40 [165812.315428] Mem-Info: [165812.323867] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:0 inactive_file:171 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560178 mapped:1547 shmem:0 pagetables:295 bounce:0 free:39323 free_pcp:0 free_cma:0 [165812.435406] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165812.574735] lowmem_reserve[]: 0 2580 15620 15620 [165812.590965] Node 0 DMA32 free:59468kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:200kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6332 all_unreclaimable? yes [165812.740473] lowmem_reserve[]: 0 0 13040 13040 [165812.755910] Node 0 Normal free:36816kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:332kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089468kB kernel_stack:14000kB pagetables:28kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:148414 all_unreclaimable? yes [165812.910693] lowmem_reserve[]: 0 0 0 0 [165812.923952] Node 1 Normal free:45456kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:12kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957104kB kernel_stack:16640kB pagetables:1148kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1040 all_unreclaimable? yes [165813.080442] lowmem_reserve[]: 0 0 0 0 [165813.093820] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165813.140495] Node 0 DMA32: 43*4kB (UEM) 1860*8kB (UEM) 1336*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59500kB [165813.191495] Node 0 Normal: 161*4kB (UEM) 146*8kB (UEM) 116*16kB (UEM) 226*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36820kB [165813.244293] Node 1 Normal: 567*4kB (UEM) 396*8kB (UEM) 242*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45468kB [165813.296016] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165813.326081] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165813.355305] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165813.385354] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165813.414542] 1787 total pagecache pages [165813.428291] 0 pages in swap cache [165813.440602] Swap cache stats: add 314903, delete 314903, find 93792/151457 [165813.464657] Free swap = 16277596kB [165813.477540] Total swap = 16319484kB [165813.490414] 8369063 pages RAM [165813.501565] 0 pages HighMem/MovableOnly [165813.515568] 241261 pages reserved [165813.527857] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165813.555044] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165813.584816] [ 1046] 0 1046 1588718 3690 132 0 -1000 multipathd [165813.613730] [ 2373] 0 2373 13863 110 27 124 -1000 auditd [165813.641486] [ 2444] 81 2444 9196 198 21 151 -900 dbus-daemon [165813.670667] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165813.697871] [67651] 0 67651 79 0 3 7 0 systemd-cgroups [165813.728171] [67657] 0 67657 76 0 2 2 0 systemd-cgroups [165813.758439] Out of memory: Kill process 67651 (systemd-cgroups) score 0 or sacrifice child [165813.787001] Killed process 67651 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [165813.822040] systemd-cgroups invoked oom-killer: gfp_mask=0x84d0, order=0, oom_score_adj=0 [165813.850301] systemd-cgroups cpuset=/ mems_allowed=0-1 [165813.868233] CPU: 4 PID: 67657 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165813.908365] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165813.946617] ffff88042d0c6eb0 00000000c739d5f5 ffff88082e6376f0 ffffffff816a3e2d [165813.972324] ffff88082e637780 ffffffff8169f246 ffffffff810e93ac ffff8807ebebe750 [165813.998007] ffff8807ebebe768 0000000000000202 0000000000000202 ffff88082e637770 [165814.023692] Call Trace: [165814.032983] [] dump_stack+0x19/0x1b [165814.051141] [] dump_header+0x90/0x229 [165814.069866] [] ? ktime_get_ts64+0x4c/0xf0 [165814.089708] [] ? delayacct_end+0x8f/0xb0 [165814.109241] [] oom_kill_process+0x254/0x3d0 [165814.129609] [] ? oom_unkillable_task+0xcd/0x120 [165814.151122] [] ? find_lock_task_mm+0x56/0xc0 [165814.171724] [] out_of_memory+0x4b6/0x4f0 [165814.191188] [] __alloc_pages_slowpath+0x5d6/0x724 [165814.213212] [] __alloc_pages_nodemask+0x405/0x420 [165814.235215] [] alloc_pages_current+0x98/0x110 [165814.256049] [] __pmd_alloc+0x24/0x120 [165814.274599] [] handle_mm_fault+0xde9/0xfa0 [165814.294535] [] ? __vma_link_file+0x48/0x70 [165814.314451] [] ? vma_link+0x7d/0xc0 [165814.332339] [] ? vma_set_page_prot+0x3f/0x60 [165814.352779] [] ? mmap_region+0x1ba/0x640 [165814.372055] [] __do_page_fault+0x154/0x450 [165814.391912] [] do_page_fault+0x35/0x90 [165814.410624] [] page_fault+0x28/0x30 [165814.428486] [] ? __clear_user+0x25/0x50 [165814.447481] [] clear_user+0x30/0x40 [165814.465331] [] padzero+0x29/0x40 [165814.482312] [] load_elf_interp.constprop.10+0x365/0x409 [165814.505877] [] load_elf_binary+0x92d/0xe00 [165814.525724] [] ? load_elf_library+0x220/0x220 [165814.546430] [] search_binary_handler+0xed/0x300 [165814.567711] [] do_execve_common.isra.25+0x5b6/0x6c0 [165814.590143] [] do_execve+0x18/0x20 [165814.607702] [] ____call_usermodehelper+0xfc/0x130 [165814.629545] [] ? ____call_usermodehelper+0x130/0x130 [165814.652248] [] call_helper+0x1e/0x20 [165814.670369] [] ret_from_fork+0x58/0x90 [165814.689061] [] ? ____call_usermodehelper+0x130/0x130 [165814.711783] Mem-Info: [165814.720176] active_anon:0 inactive_anon:4 isolated_anon:0 active_file:58 inactive_file:92 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560178 mapped:1547 shmem:0 pagetables:295 bounce:0 free:39330 free_pcp:0 free_cma:0 [165814.831314] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165814.970325] lowmem_reserve[]: 0 2580 15620 15620 [165814.986504] Node 0 DMA32 free:59376kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:16kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:21732 all_unreclaimable? yes [165815.137364] lowmem_reserve[]: 0 0 13040 13040 [165815.152767] Node 0 Normal free:36860kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:52kB inactive_file:256kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089460kB kernel_stack:14000kB pagetables:16kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:372 all_unreclaimable? yes [165815.306789] lowmem_reserve[]: 0 0 0 0 [165815.320002] Node 1 Normal free:45484kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4352kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957104kB kernel_stack:16624kB pagetables:1148kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1274 all_unreclaimable? yes [165815.475839] lowmem_reserve[]: 0 0 0 0 [165815.489136] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165815.535674] Node 0 DMA32: 41*4kB (UEM) 1860*8kB (UEM) 1336*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59492kB [165815.586593] Node 0 Normal: 169*4kB (UEM) 147*8kB (UEM) 116*16kB (UEM) 226*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36860kB [165815.639312] Node 1 Normal: 567*4kB (UEM) 396*8kB (UEM) 243*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45484kB [165815.690956] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165815.720980] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165815.750153] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165815.780162] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165815.809308] 1779 total pagecache pages [165815.822979] 0 pages in swap cache [165815.835242] Swap cache stats: add 314907, delete 314907, find 93792/151457 [165815.859242] Free swap = 16277612kB [165815.872113] Total swap = 16319484kB [165815.884951] 8369063 pages RAM [165815.896059] 0 pages HighMem/MovableOnly [165815.910027] 241261 pages reserved [165815.922278] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165815.949429] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165815.979154] [ 1046] 0 1046 1588718 3741 132 0 -1000 multipathd [165816.008024] [ 2373] 0 2373 13863 110 27 124 -1000 auditd [165816.035760] [ 2444] 81 2444 9196 198 21 151 -900 dbus-daemon [165816.064913] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165816.092057] [67657] 0 67657 76 0 2 2 0 systemd-cgroups [165816.122297] Out of memory: Kill process 67657 (systemd-cgroups) score 0 or sacrifice child [165816.150849] Killed process 67657 (systemd-cgroups) total-vm:304kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [165816.343572] systemd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [165816.369801] systemd cpuset=/ mems_allowed=0-1 [165816.385421] CPU: 2 PID: 1 Comm: systemd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165816.421070] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165816.459294] ffff88017a1d0000 00000000f5e675ef ffff88017a1db988 ffffffff816a3e2d [165816.484966] ffff88017a1dba18 ffffffff8169f246 ffffffff810e93ac ffff88017a1271a0 [165816.510634] ffff88017a1271b8 0000000000000202 0000000000000202 ffff88017a1dba08 [165816.536288] Call Trace: [165816.545567] [] dump_stack+0x19/0x1b [165816.563703] [] dump_header+0x90/0x229 [165816.582387] [] ? ktime_get_ts64+0x4c/0xf0 [165816.602198] [] ? delayacct_end+0x8f/0xb0 [165816.621717] [] oom_kill_process+0x254/0x3d0 [165816.642083] [] ? oom_unkillable_task+0xcd/0x120 [165816.663594] [] ? find_lock_task_mm+0x56/0xc0 [165816.684200] [] out_of_memory+0x4b6/0x4f0 [165816.703638] [] __alloc_pages_slowpath+0x5d6/0x724 [165816.725634] [] __alloc_pages_nodemask+0x405/0x420 [165816.747600] [] alloc_pages_vma+0xb5/0x200 [165816.767284] [] read_swap_cache_async+0xed/0x160 [165816.788671] [] ? task_numa_fault+0x8d0/0xbb0 [165816.809175] [] swapin_readahead+0xa8/0x110 [165816.829086] [] handle_mm_fault+0xadb/0xfa0 [165816.848981] [] __do_page_fault+0x154/0x450 [165816.868845] [] do_page_fault+0x35/0x90 [165816.887544] [] page_fault+0x28/0x30 [165816.905361] Mem-Info: [165816.913719] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:241 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560176 mapped:1547 shmem:0 pagetables:292 bounce:0 free:39363 free_pcp:0 free_cma:0 [165817.024865] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165817.164151] lowmem_reserve[]: 0 2580 15620 15620 [165817.180411] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:48kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:38834 all_unreclaimable? no [165817.329776] lowmem_reserve[]: 0 0 13040 13040 [165817.345258] Node 0 Normal free:36908kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:552kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089460kB kernel_stack:13984kB pagetables:16kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:17615 all_unreclaimable? yes [165817.499839] lowmem_reserve[]: 0 0 0 0 [165817.513084] Node 1 Normal free:45484kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957104kB kernel_stack:16624kB pagetables:1148kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:910 all_unreclaimable? yes [165817.668662] lowmem_reserve[]: 0 0 0 0 [165817.681939] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165817.728428] Node 0 DMA32: 42*4kB (UEM) 1860*8kB (UEM) 1336*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165817.779308] Node 0 Normal: 169*4kB (UEM) 147*8kB (UEM) 117*16kB (UEM) 226*32kB (UEM) 385*64kB (UEM) 8*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36876kB [165817.831982] Node 1 Normal: 567*4kB (UEM) 398*8kB (UEM) 244*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45516kB [165817.883589] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165817.913620] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165817.942764] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165817.972763] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165818.001889] 1771 total pagecache pages [165818.015568] 0 pages in swap cache [165818.027815] Swap cache stats: add 314907, delete 314907, find 93792/151457 [165818.051802] Free swap = 16277620kB [165818.064643] Total swap = 16319484kB [165818.077489] 8369063 pages RAM [165818.088605] 0 pages HighMem/MovableOnly [165818.102575] 241261 pages reserved [165818.114822] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165818.142009] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165818.171735] [ 1046] 0 1046 1588718 3704 132 0 -1000 multipathd [165818.200609] [ 2373] 0 2373 13863 110 27 124 -1000 auditd [165818.228329] [ 2444] 81 2444 9196 198 21 151 -900 dbus-daemon [165818.257497] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165818.284659] Out of memory: Kill process 2444 (dbus-daemon) score 0 or sacrifice child [165818.311795] Killed process 2444 (dbus-daemon) total-vm:36784kB, anon-rss:0kB, file-rss:792kB, shmem-rss:0kB [165818.372072] auditd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=-1000 [165818.399197] auditd cpuset=/ mems_allowed=0-1 [165818.414564] CPU: 17 PID: 2373 Comm: auditd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165818.451124] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165818.489389] ffff88082c1caf70 00000000689d2a54 ffff88082ae7fa00 ffffffff816a3e2d [165818.515098] ffff88082ae7fa90 ffffffff8169f246 ffffffff8113d71e ffff88082ae7fa28 [165818.540791] ffffffff811142f1 ffff88082ae7fa68 ffffffff81185e4d ffffffff81185ef6 [165818.566468] Call Trace: [165818.575769] [] dump_stack+0x19/0x1b [165818.593934] [] dump_header+0x90/0x229 [165818.612634] [] ? __delayacct_freepages_end+0x2e/0x30 [165818.635614] [] ? cpuset_mems_allowed_intersects+0x21/0x30 [165818.660006] [] ? oom_unkillable_task+0xcd/0x120 [165818.681537] [] ? find_lock_task_mm+0x56/0xc0 [165818.702188] [] out_of_memory+0x4dc/0x4f0 [165818.721665] [] __alloc_pages_slowpath+0x5d6/0x724 [165818.743721] [] __alloc_pages_nodemask+0x405/0x420 [165818.765740] [] alloc_pages_vma+0xb5/0x200 [165818.785451] [] ? end_swap_bio_write+0x80/0x80 [165818.806331] [] read_swap_cache_async+0xed/0x160 [165818.827752] [] ? mem_cgroup_update_page_stat+0x16/0x50 [165818.851171] [] swapin_readahead+0xa8/0x110 [165818.871143] [] handle_mm_fault+0xadb/0xfa0 [165818.891066] [] __do_page_fault+0x154/0x450 [165818.910993] [] do_page_fault+0x35/0x90 [165818.929724] [] page_fault+0x28/0x30 [165818.947586] Mem-Info: [165818.956000] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:46 inactive_file:195 isolated_file:0 unevictable:6774 dirty:0 writeback:0 unstable:0 slab_reclaimable:12081 slab_unreclaimable:560174 mapped:1547 shmem:0 pagetables:271 bounce:0 free:39392 free_pcp:0 free_cma:0 [165819.067678] Node 0 DMA free:15580kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [165819.207031] lowmem_reserve[]: 0 2580 15620 15620 [165819.223308] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:112kB inactive_file:0kB unevictable:332kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:332kB dirty:0kB writeback:0kB mapped:340kB shmem:0kB slab_reclaimable:3100kB slab_unreclaimable:194140kB kernel_stack:1584kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:66338 all_unreclaimable? yes [165819.373319] lowmem_reserve[]: 0 0 13040 13040 [165819.388832] Node 0 Normal free:36912kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:500kB unevictable:1684kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:1684kB dirty:0kB writeback:0kB mapped:1532kB shmem:0kB slab_reclaimable:19392kB slab_unreclaimable:1089460kB kernel_stack:13984kB pagetables:16kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:772 all_unreclaimable? yes [165819.542952] lowmem_reserve[]: 0 0 0 0 [165819.556229] Node 1 Normal free:45596kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:232kB inactive_file:364kB unevictable:25080kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:25080kB dirty:0kB writeback:0kB mapped:4316kB shmem:0kB slab_reclaimable:25832kB slab_unreclaimable:957096kB kernel_stack:16624kB pagetables:1064kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:910 all_unreclaimable? yes [165819.711879] lowmem_reserve[]: 0 0 0 0 [165819.725169] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15580kB [165819.771714] Node 0 DMA32: 42*4kB (UEM) 1860*8kB (UEM) 1336*16kB (UE) 501*32kB (UEM) 104*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59496kB [165819.822587] Node 0 Normal: 157*4kB (UEM) 145*8kB (UEM) 115*16kB (UEM) 224*32kB (UEM) 385*64kB (UEM) 9*128kB (UEM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36844kB [165819.875300] Node 1 Normal: 589*4kB (UEM) 401*8kB (UEM) 244*16kB (UEM) 352*32kB (UEM) 351*64kB (UEM) 19*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45628kB [165819.926872] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165819.956978] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165819.986124] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [165820.016172] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [165820.045305] 1770 total pagecache pages [165820.058991] 0 pages in swap cache [165820.071239] Swap cache stats: add 314911, delete 314911, find 93794/151459 [165820.095231] Free swap = 16278196kB [165820.108069] Total swap = 16319484kB [165820.120959] 8369063 pages RAM [165820.132071] 0 pages HighMem/MovableOnly [165820.146041] 241261 pages reserved [165820.158289] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [165820.185451] [ 1041] 0 1041 11285 232 22 428 -1000 systemd-udevd [165820.215185] [ 1046] 0 1046 1588718 3688 132 0 -1000 multipathd [165820.244070] [ 2373] 0 2373 13863 110 27 124 -1000 auditd [165820.271790] [ 3695] 0 3695 26499 278 58 244 -1000 sshd [165820.298970] Kernel panic - not syncing: Out of memory and no killable processes... [165820.331453] CPU: 17 PID: 2373 Comm: auditd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [165820.368042] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [165820.406347] 0000000000000000 00000000689d2a54 ffff88082ae7fa10 ffffffff816a3e2d [165820.432105] ffff88082ae7fa90 ffffffff8169dd14 ffffffff00000008 ffff88082ae7faa0 [165820.457850] ffff88082ae7fa40 00000000689d2a54 0000000000000116 0000000000000000 [165820.483653] Call Trace: [165820.493025] [] dump_stack+0x19/0x1b [165820.511236] [] panic+0xe8/0x20d [165820.528278] [] out_of_memory+0x4ea/0x4f0 [165820.547876] [] __alloc_pages_slowpath+0x5d6/0x724 [165820.570048] [] __alloc_pages_nodemask+0x405/0x420 [165820.592178] [] alloc_pages_vma+0xb5/0x200 [165820.612040] [] ? end_swap_bio_write+0x80/0x80 [165820.632982] [] read_swap_cache_async+0xed/0x160 [165820.654499] [] ? mem_cgroup_update_page_stat+0x16/0x50 [165820.678009] [] swapin_readahead+0xa8/0x110 [165820.698056] [] handle_mm_fault+0xadb/0xfa0 [165820.718094] [] __do_page_fault+0x154/0x450 [165820.738096] [] do_page_fault+0x35/0x90 [165820.756929] [] page_fault+0x28/0x30