Mar 9 11:22:27 lustre-utils01 kernel: [108887.580788] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.580802] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.580803] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.580838] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.580841] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.580848] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.580855] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.580857] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.580870] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.580912] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 11:22:27 lustre-utils01 kernel: [108887.580922] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 11:22:27 lustre-utils01 kernel: [108887.580925] [] kmem_cache_alloc_trace+0x132/0x140 Mar 9 11:22:27 lustre-utils01 kernel: [108887.580934] [] ? lnet_md_alloc.isra.2+0x74/0x280 [lnet] Mar 9 11:22:27 lustre-utils01 kernel: [108887.580941] [] LNetPut+0x122/0x900 [lnet] Mar 9 11:22:27 lustre-utils01 kernel: [108887.580975] [] ? cfs_percpt_unlock+0x42/0xd0 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581071] [] ptl_send_buf+0x12f/0x5d0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581079] [] ? LNetMDAttach+0x2db/0x370 [lnet] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581097] [] ? read_tsc+0x9/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581124] [] ptl_send_rpc+0x522/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581144] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581166] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581197] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581219] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581240] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581255] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581277] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.581287] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581290] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581301] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581304] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581305] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.581306] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.581309] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581310] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.581311] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581312] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.581313] CPU 0: hi: 186, btch: 31 usd: 144 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581316] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581316] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581316] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581316] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581316] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581316] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581319] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.581324] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581326] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.581342] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581344] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.581348] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581350] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.581357] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.581365] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.581370] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.581371] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.581373] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.581374] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.581374] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.633902] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108887.633907] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108887.633907] 80602 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.633908] 4048542 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.633913] SLUB: Unable to allocate memory on node -1 (gfp=0x40) Mar 9 11:22:27 lustre-utils01 kernel: [108887.633915] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.633917] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634011] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: out of memory at /var/lib/dkms/lustre-client-modules-2.5.0/2.5.0/build/lnet/include/lnet/lib-lnet.h:506 (tried to alloc '(msg)' = 352) Mar 9 11:22:27 lustre-utils01 kernel: [108887.634248] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: 757787183 total bytes allocated by lnet Mar 9 11:22:27 lustre-utils01 kernel: [108887.634360] LNetError: 2001:0:(lib-move.c:2138:LNetPut()) Dropping PUT to 12345-172.17.117.148@tcp: ENOMEM on lnet_msg_t Mar 9 11:22:27 lustre-utils01 kernel: [108887.634482] LustreError: 2001:0:(niobuf.c:96:ptl_send_buf()) LNetPut(12345-172.17.117.148@tcp, 17, 1461982792553984) failed: -12 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634641] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1394363770/real 1394363770] req@ffff8803d09a9800 x1461982792553984/t0(0) o103->lus13-OST0014-osc-ffff880412525000@172.17.117.148@tcp:17/18 lens 344/224 e 0 to 1 dl 1394364000 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634665] Lustre: lus13-OST0014-osc-ffff880412525000: Connection to lus13-OST0014 (at 172.17.117.148@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 11:22:27 lustre-utils01 kernel: [108887.634691] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634694] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.634695] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.634706] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634708] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634720] [] ? console_unlock.part.8+0x22e/0x280 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634725] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634731] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634733] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634737] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634802] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.634810] [] ? check_preempt_curr+0x84/0xa0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634812] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634814] [] ? ttwu_do_activate.constprop.86+0x66/0x70 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634836] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.634863] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.634903] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.634926] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.634930] [] ? default_wake_function+0x12/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108887.634950] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.634972] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.634991] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.635013] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.635032] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.635053] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.635056] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635077] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.635080] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635083] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635086] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635088] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635090] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.635091] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.635093] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635094] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.635096] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635096] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.635098] CPU 0: hi: 186, btch: 31 usd: 144 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635100] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635100] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635100] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635100] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635100] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635100] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635103] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.635107] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635109] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.635113] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635115] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.635118] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635120] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.635128] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.635135] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.635141] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.635142] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.635143] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.635144] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.635145] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.682950] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108887.682954] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108887.682955] 80602 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.682955] 4048542 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.682961] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108887.682963] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.682965] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683132] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff8803e20b4800 x1461982792553992/t0(0) o103->lus13-OST0027-osc-ffff880412525000@172.17.117.151@tcp:17/18 lens 344/224 e 0 to 1 dl 0 ref 1 fl Rpc:EX/0/ffffffff rc -12/-1 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683163] Lustre: lus13-OST0027-osc-ffff880412525000: Connection to lus13-OST0027 (at 172.17.117.151@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 11:22:27 lustre-utils01 kernel: [108887.683187] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683189] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.683191] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.683201] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683204] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683210] [] ? console_unlock.part.8+0x22e/0x280 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683215] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683222] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683224] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683228] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683298] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683322] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683326] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683351] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683374] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683394] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683415] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683419] [] ? default_wake_function+0x12/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683439] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683460] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683479] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683501] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683520] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683542] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683544] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683565] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.683568] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683571] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683574] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683576] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683578] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.683579] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.683581] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683582] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.683583] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683584] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.683586] CPU 0: hi: 186, btch: 31 usd: 144 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683588] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683588] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683588] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683588] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683588] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683588] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683591] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.683595] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683598] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.683619] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683621] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.683626] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683628] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.683636] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.683643] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.683648] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.683650] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.683652] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.683653] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.683654] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.731179] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108887.731184] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108887.731185] 80602 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.731185] 4048542 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.731191] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108887.731193] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731195] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731484] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731488] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.731490] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.731503] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731540] [] ? number.isra.2+0x355/0x390 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731545] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731549] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731552] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731580] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731586] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731589] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731591] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731595] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731641] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731655] [] ? LNetPut+0x320/0x900 [lnet] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731658] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731678] [] ? ptl_send_buf+0x12f/0x5d0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731699] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731721] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731741] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731762] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731782] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731803] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731823] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731844] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731864] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731885] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731902] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731925] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.731929] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731931] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731935] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731937] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731938] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.731940] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.731942] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731943] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.731944] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731945] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.731946] CPU 0: hi: 186, btch: 31 usd: 141 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731950] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731950] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731950] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731950] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731950] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731950] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731953] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.731957] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731960] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.731963] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731965] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.731969] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.731971] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.731978] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.731994] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.732000] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.732001] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.732002] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.732003] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.732004] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.779557] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108887.779563] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108887.779564] 80604 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.779565] 4048543 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.779570] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108887.779572] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779574] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779676] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779679] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.779680] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.779691] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779695] [] ? number.isra.2+0x355/0x390 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779697] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779724] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779726] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779750] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779753] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779756] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779758] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779761] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779806] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779813] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779815] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779837] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779859] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779879] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779900] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779904] [] ? default_wake_function+0x12/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108887.779924] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779953] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.779985] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.780008] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.780027] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.780049] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.780051] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780073] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.780076] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780078] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780082] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780084] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780085] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.780087] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.780089] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780090] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.780091] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780092] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.780093] CPU 0: hi: 186, btch: 31 usd: 141 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780097] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780097] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780097] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780097] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780097] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780097] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780100] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.780104] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780107] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.780110] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780112] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.780123] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780126] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.780134] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.780141] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.780147] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.780148] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.780150] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.780151] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.780151] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.827729] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108887.827733] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108887.827734] 80604 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.827735] 4048543 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.827740] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108887.827742] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827744] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827838] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827841] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.827842] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.827852] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827857] [] ? number.isra.2+0x355/0x390 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827859] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827861] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827863] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827886] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.827889] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827892] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827894] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827897] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827948] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.827956] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.827959] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108887.827980] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828002] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828022] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828044] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828047] [] ? default_wake_function+0x12/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828067] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828088] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828108] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828136] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828159] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828181] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828184] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828206] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.828209] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828211] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828215] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828217] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828218] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.828220] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.828222] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828223] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.828224] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828225] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.828227] CPU 0: hi: 186, btch: 31 usd: 141 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828230] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828230] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828230] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828230] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828230] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828230] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828233] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.828258] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828261] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.828266] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828268] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.828272] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828273] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.828281] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.828289] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.828294] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.828295] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.828297] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.828298] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.828298] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.875914] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108887.875918] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108887.875919] 80604 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.875920] 4048543 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.875925] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108887.875927] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.875928] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876023] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876026] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.876028] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.876038] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876043] [] ? number.isra.2+0x355/0x390 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876045] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876047] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876049] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876076] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876081] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876085] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876087] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876092] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876137] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876144] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876147] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876169] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876190] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876210] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876231] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876260] [] ? default_wake_function+0x12/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876281] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876303] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876323] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876344] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876363] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876385] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876387] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876409] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.876412] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876415] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876418] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876420] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876422] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.876423] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.876425] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876426] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.876427] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876428] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.876430] CPU 0: hi: 186, btch: 31 usd: 141 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876433] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876433] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876433] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876433] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876433] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876433] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876436] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.876440] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876443] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.876447] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876449] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.876452] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876454] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.876462] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.876469] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.876474] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.876476] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.876477] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.876478] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.876479] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.924239] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108887.924243] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108887.924244] 80604 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.924245] 4048543 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.924250] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108887.924277] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924279] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924456] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924460] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.924461] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.924472] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924478] [] ? number.isra.2+0x355/0x390 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924480] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924484] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924487] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924514] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924521] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924527] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924532] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924537] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924583] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924591] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924593] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924615] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924637] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924656] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924678] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924682] [] ? default_wake_function+0x12/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924701] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924723] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924742] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924772] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924794] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924815] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924818] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924840] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.924843] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924845] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924849] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924851] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924852] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.924854] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.924856] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924857] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.924858] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924859] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.924860] CPU 0: hi: 186, btch: 31 usd: 142 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924863] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924863] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924863] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924863] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924863] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924863] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924866] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.924871] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924873] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.924877] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924889] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.924893] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924895] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.924903] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.924910] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.924915] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.924917] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.924918] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.924919] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.924920] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.972650] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108887.972655] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108887.972656] 80604 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.972657] 4048542 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108887.972663] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108887.972665] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972667] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972812] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972815] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108887.972817] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108887.972827] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972832] [] ? number.isra.2+0x355/0x390 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972834] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972836] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972839] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972863] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.972867] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972870] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972872] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972875] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972919] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.972926] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108887.972928] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108887.972950] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.972972] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.972992] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973013] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973017] [] ? default_wake_function+0x12/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973037] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973059] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973079] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973100] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973119] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973141] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973143] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973165] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108887.973168] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973170] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973174] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973176] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973177] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108887.973179] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.973197] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973198] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.973199] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973200] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108887.973201] CPU 0: hi: 186, btch: 31 usd: 142 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973205] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973205] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973205] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973205] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973205] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973205] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973208] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108887.973212] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973215] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.973218] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973220] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108887.973224] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973225] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.973233] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.973241] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.973246] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108887.973248] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108887.973249] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108887.973250] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108887.973251] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108888.020674] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108888.020678] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108888.020679] 80604 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108888.020680] 4048542 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108888.020685] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108888.020687] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020689] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020773] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020776] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 11:22:27 lustre-utils01 kernel: [108888.020777] Call Trace: Mar 9 11:22:27 lustre-utils01 kernel: [108888.020787] [] warn_alloc_failed+0xf6/0x150 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020792] [] ? number.isra.2+0x355/0x390 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020793] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020796] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020797] [] ? put_dec+0x72/0x90 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020820] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108888.020824] [] alloc_pages_current+0xba/0x170 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020827] [] allocate_slab+0x190/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020828] [] new_slab+0x30/0x1a0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020849] [] __slab_alloc+0x17d/0x2b2 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020893] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.020901] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 11:22:27 lustre-utils01 kernel: [108888.020903] [] __kmalloc+0x162/0x190 Mar 9 11:22:27 lustre-utils01 kernel: [108888.020925] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.020947] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.020967] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.020988] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.020992] [] ? default_wake_function+0x12/0x20 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021012] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.021033] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.021053] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.021074] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.021094] [] ? ptlrpc_set_next_timeout+0x66/0x1a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.021115] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.021118] [] ? try_to_wake_up+0x200/0x200 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021139] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 11:22:27 lustre-utils01 kernel: [108888.021142] [] kthread+0xc0/0xd0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021144] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021148] [] ret_from_fork+0x7c/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021150] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021151] Mem-Info: Mar 9 11:22:27 lustre-utils01 kernel: [108888.021152] Node 0 DMA per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108888.021155] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021155] Node 0 DMA32 per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108888.021157] CPU 0: hi: 186, btch: 31 usd: 127 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021158] Node 0 Normal per-cpu: Mar 9 11:22:27 lustre-utils01 kernel: [108888.021159] CPU 0: hi: 186, btch: 31 usd: 142 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021162] active_anon:6329 inactive_anon:29353 isolated_anon:0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021162] active_file:22920 inactive_file:22441 isolated_file:0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021162] unevictable:7965 dirty:1 writeback:0 unstable:0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021162] free:26023 slab_reclaimable:32389 slab_unreclaimable:3756556 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021162] mapped:8422 shmem:33 pagetables:1246 bounce:0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021162] free_cma:0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021165] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 11:22:27 lustre-utils01 kernel: [108888.021169] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021172] Node 0 DMA32 free:47104kB min:12604kB low:15752kB high:18904kB active_anon:524kB inactive_anon:8824kB active_file:3484kB inactive_file:1440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:4kB writeback:0kB mapped:44kB shmem:0kB slab_reclaimable:8288kB slab_unreclaimable:2979844kB kernel_stack:40kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108888.021176] lowmem_reserve[]: 0 0 13104 13104 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021178] Node 0 Normal free:41084kB min:54912kB low:68640kB high:82368kB active_anon:24792kB inactive_anon:108588kB active_file:88196kB inactive_file:88324kB unevictable:31860kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:31860kB dirty:0kB writeback:0kB mapped:33644kB shmem:132kB slab_reclaimable:121268kB slab_unreclaimable:12046376kB kernel_stack:2184kB pagetables:4968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 11:22:27 lustre-utils01 kernel: [108888.021181] lowmem_reserve[]: 0 0 0 0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021183] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 11:22:27 lustre-utils01 kernel: [108888.021191] Node 0 DMA32: 924*4kB (UEM) 256*8kB (UM) 147*16kB (UM) 224*32kB (UEM) 389*64kB (UM) 18*128kB (UEM) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 47072kB Mar 9 11:22:27 lustre-utils01 kernel: [108888.021198] Node 0 Normal: 10271*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41084kB Mar 9 11:22:27 lustre-utils01 kernel: [108888.021208] 47613 total pagecache pages Mar 9 11:22:27 lustre-utils01 kernel: [108888.021210] 1359 pages in swap cache Mar 9 11:22:27 lustre-utils01 kernel: [108888.021211] Swap cache stats: add 13656, delete 12297, find 7232/7421 Mar 9 11:22:27 lustre-utils01 kernel: [108888.021212] Free swap = 38744964kB Mar 9 11:22:27 lustre-utils01 kernel: [108888.021213] Total swap = 38793204kB Mar 9 11:22:27 lustre-utils01 kernel: [108888.068654] 4194288 pages RAM Mar 9 11:22:27 lustre-utils01 kernel: [108888.068658] 85481 pages reserved Mar 9 11:22:27 lustre-utils01 kernel: [108888.068659] 80604 pages shared Mar 9 11:22:27 lustre-utils01 kernel: [108888.068659] 4048542 pages non-shared Mar 9 11:22:27 lustre-utils01 kernel: [108888.068664] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 11:22:27 lustre-utils01 kernel: [108888.068666] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.068668] node 0: slabs: 79218, objs: 633744, free: 0 Mar 9 11:22:27 lustre-utils01 kernel: [108888.070038] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: out of memory at /var/lib/dkms/lustre-client-modules-2.5.0/2.5.0/build/lnet/include/lnet/lib-lnet.h:506 (tried to alloc '(msg)' = 352) Mar 9 11:22:27 lustre-utils01 kernel: [108888.070240] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: 757789295 total bytes allocated by lnet Mar 9 11:22:27 lustre-utils01 kernel: [108888.070348] LNetError: 2001:0:(lib-move.c:2138:LNetPut()) Dropping PUT to 12345-172.17.117.146@tcp: ENOMEM on lnet_msg_t Mar 9 11:22:27 lustre-utils01 kernel: [108888.070483] LustreError: 2001:0:(niobuf.c:96:ptl_send_buf()) LNetPut(12345-172.17.117.146@tcp, 28, 1461982792556272) failed: -12 Mar 9 11:22:27 lustre-utils01 kernel: [108888.278266] Lustre: lus13-MDT0000-mdc-ffff880412525000: Connection restored to lus13-MDT0000 (at 172.17.117.153@tcp) Mar 9 11:22:27 lustre-utils01 kernel: [108992.910497] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394363771/real 0] req@ffff8803d74a1800 x1461982792617828/t0(0) o8->lus13-OST0000-osc-ffff880412525000@172.17.117.144@tcp:28/4 lens 400/544 e 0 to 1 dl 1394363876 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 11:22:27 lustre-utils01 kernel: [108992.910511] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 39 previous similar messages Mar 9 11:22:27 lustre-utils01 kernel: [109066.087234] Lustre: lus13-OST0000-osc-ffff880412525000: Connection restored to lus13-OST0000 (at 172.17.117.144@tcp) Mar 9 11:22:27 lustre-utils01 kernel: [109067.599580] Lustre: lus13-OST0003-osc-ffff880412525000: Connection restored to lus13-OST0003 (at 172.17.117.144@tcp) Mar 9 11:22:27 lustre-utils01 kernel: [109067.599596] Lustre: Skipped 2 previous similar messages Mar 9 11:22:27 lustre-utils01 kernel: [109069.046796] Lustre: lus13-OST0006-osc-ffff880412525000: Connection restored to lus13-OST0006 (at 172.17.117.145@tcp) Mar 9 11:22:27 lustre-utils01 kernel: [109069.046801] Lustre: Skipped 2 previous similar messages Mar 9 11:22:27 lustre-utils01 kernel: [109073.364607] Lustre: lus13-OST000b-osc-ffff880412525000: Connection restored to lus13-OST000b (at 172.17.117.146@tcp) Mar 9 11:22:27 lustre-utils01 kernel: [109073.364612] Lustre: Skipped 4 previous similar messages Mar 9 11:22:27 lustre-utils01 kernel: [109078.174338] Lustre: lus13-OST0011-osc-ffff880412525000: Connection restored to lus13-OST0011 (at 172.17.117.147@tcp) Mar 9 11:22:27 lustre-utils01 kernel: [109078.174343] Lustre: Skipped 5 previous similar messages Mar 9 11:22:27 lustre-utils01 kernel: [109086.302522] Lustre: lus13-OST001b-osc-ffff880412525000: Connection restored to lus13-OST001b (at 172.17.117.149@tcp) Mar 9 11:22:27 lustre-utils01 kernel: [109086.302558] Lustre: Skipped 9 previous similar messages Mar 9 11:24:14 lustre-utils01 kernel: [109367.718284] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394363951/real 0] req@ffff880203b48400 x1461982795991444/t0(0) o103->lus13-OST0003-osc-ffff880412525000@172.17.117.144@tcp:17/18 lens 560/224 e 0 to 1 dl 1394364250 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 11:24:14 lustre-utils01 kernel: [109367.718289] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 39 previous similar messages Mar 9 11:24:14 lustre-utils01 kernel: [109367.718391] Lustre: lus13-OST0003-osc-ffff880412525000: Connection to lus13-OST0003 (at 172.17.117.144@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 11:24:14 lustre-utils01 kernel: [109367.718393] Lustre: Skipped 39 previous similar messages Mar 9 11:24:14 lustre-utils01 kernel: [109369.724900] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394363950/real 0] req@ffff8803a6324800 x1461982793266160/t0(0) o103->lus13-OST0000-osc-ffff880412525000@172.17.117.144@tcp:17/18 lens 344/224 e 0 to 1 dl 1394364249 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 11:24:14 lustre-utils01 kernel: [109369.724905] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 28122 previous similar messages Mar 9 11:24:21 lustre-utils01 kernel: [109371.496988] Lustre: lus13-OST0008-osc-ffff880412525000: Connection to lus13-OST0008 (at 172.17.117.145@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 11:24:21 lustre-utils01 kernel: [109371.496993] Lustre: Skipped 4 previous similar messages Mar 9 11:24:21 lustre-utils01 kernel: [109373.731017] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394363954/real 0] req@ffff880147147400 x1461982794514664/t0(0) o103->lus13-OST0004-osc-ffff880412525000@172.17.117.144@tcp:17/18 lens 352/224 e 0 to 1 dl 1394364253 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 11:24:21 lustre-utils01 kernel: [109373.731022] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 46320 previous similar messages Mar 9 11:24:30 lustre-utils01 kernel: [109378.961097] Lustre: lus13-OST000e-osc-ffff880412525000: Connection to lus13-OST000e (at 172.17.117.146@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 11:24:30 lustre-utils01 kernel: [109378.961103] Lustre: Skipped 4 previous similar messages Mar 9 11:24:30 lustre-utils01 kernel: [109381.735230] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394363953/real 0] req@ffff880256477400 x1461982795212920/t0(0) o103->lus13-OST0006-osc-ffff880412525000@172.17.117.145@tcp:17/18 lens 360/224 e 0 to 1 dl 1394364252 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 11:24:30 lustre-utils01 kernel: [109381.735235] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 127085 previous similar messages Mar 9 11:24:37 lustre-utils01 kernel: [109387.842420] Lustre: lus13-OST0018-osc-ffff880412525000: Connection to lus13-OST0018 (at 172.17.117.148@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 11:24:37 lustre-utils01 kernel: [109387.842424] Lustre: Skipped 12 previous similar messages Mar 9 11:24:44 lustre-utils01 kernel: [109396.915803] Lustre: lus13-OST001f-osc-ffff880412525000: Connection to lus13-OST001f (at 172.17.117.150@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 11:24:44 lustre-utils01 kernel: [109396.915808] Lustre: Skipped 13 previous similar messages Mar 9 11:24:44 lustre-utils01 kernel: [109397.735686] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394363971/real 0] req@ffff8803d8a55800 x1461982793437864/t0(0) o103->lus13-OST001a-osc-ffff880412525000@172.17.117.149@tcp:17/18 lens 344/224 e 0 to 1 dl 1394364270 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 11:24:44 lustre-utils01 kernel: [109397.735692] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 360580 previous similar messages Mar 9 11:25:20 lustre-utils01 kernel: [109437.057300] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394364021/real 0] req@ffff88015d03f000 x1461982796016224/t0(0) o400->lus13-OST001a-osc-ffff880412525000@172.17.117.149@tcp:28/4 lens 224/224 e 0 to 1 dl 1394364320 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 11:25:22 lustre-utils01 kernel: [109437.057307] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 206722 previous similar messages Mar 9 11:25:36 lustre-utils01 kernel: [109452.754523] Lustre: lus13-OST0000-osc-ffff880412525000: Connection restored to lus13-OST0000 (at 172.17.117.144@tcp) Mar 9 11:25:36 lustre-utils01 kernel: [109452.754528] Lustre: Skipped 12 previous similar messages Mar 9 11:25:36 lustre-utils01 kernel: [109453.081481] LustreError: 167-0: lus13-OST0003-osc-ffff880412525000: This client was evicted by lus13-OST0003; in progress operations using this service will fail. Mar 9 11:25:36 lustre-utils01 kernel: [109453.085662] LustreError: 167-0: lus13-OST0004-osc-ffff880412525000: This client was evicted by lus13-OST0004; in progress operations using this service will fail. Mar 9 11:25:41 lustre-utils01 kernel: [109457.208253] LustreError: 11-0: lus13-OST0009-osc-ffff880412525000: Communicating with 172.17.117.145@tcp, operation obd_ping failed with -107. Mar 9 11:25:45 lustre-utils01 kernel: [109457.855412] LustreError: 167-0: lus13-OST0009-osc-ffff880412525000: This client was evicted by lus13-OST0009; in progress operations using this service will fail. Mar 9 11:25:45 lustre-utils01 kernel: [109460.138325] LustreError: 7187:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff8803c6ffe948[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100090000:0x7266a0:0x0] { Mar 9 11:25:45 lustre-utils01 kernel: [109460.138564] LustreError: 7187:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999db40: [23 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:25:45 lustre-utils01 kernel: [109460.138792] LustreError: 7187:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff88024e3f85c0: ffff8803b4335a00 0x20000041001 0x54dedc2b107ce346 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332212 blocks: 0 Mar 9 11:25:45 lustre-utils01 kernel: [109460.139037] LustreError: 7187:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff8803c6ffe948 Mar 9 11:25:45 lustre-utils01 kernel: [109460.139168] LustreError: 7187:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:25:45 lustre-utils01 kernel: [109460.139364] LustreError: 7187:0:(ldlm_resource.c:809:ldlm_resource_complain()) lus13-OST0009-osc-ffff880412525000: namespace resource [0x7266a0:0x0:0x0].0 (ffff880326f95000) refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 9 11:25:45 lustre-utils01 kernel: [109460.139626] LustreError: 7187:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x7266a0:0x0:0x0].0 (ffff880326f95000) refcount = 2 Mar 9 11:25:45 lustre-utils01 kernel: [109460.139795] LustreError: 7187:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:25:45 lustre-utils01 kernel: [109460.139949] LustreError: 7187:0:(ldlm_resource.c:1454:ldlm_resource_dump()) ### ### ns: lus13-OST0009-osc-ffff880412525000 lock: ffff8803b4335a00/0x54dedc2b107ce346 lrc: 3/1,0 mode: PR/PR res: [0x7266a0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x126400000000 nid: local remote: 0xe3af3d419fa0970 expref: -99 pid: 5437 timeout: 0 lvb_type: 1 Mar 9 11:25:51 lustre-utils01 kernel: [109467.551737] LustreError: 11-0: lus13-OST000b-osc-ffff880412525000: Communicating with 172.17.117.146@tcp, operation obd_ping failed with -107. Mar 9 11:25:53 lustre-utils01 kernel: [109467.551953] LustreError: Skipped 2 previous similar messages Mar 9 11:25:53 lustre-utils01 kernel: [109468.250584] LustreError: 11-0: lus13-OST000c-osc-ffff880412525000: Communicating with 172.17.117.146@tcp, operation obd_ping failed with -107. Mar 9 11:25:53 lustre-utils01 kernel: [109468.250767] LustreError: Skipped 3 previous similar messages Mar 9 11:25:53 lustre-utils01 kernel: [109468.696393] LustreError: 167-0: lus13-OST000b-osc-ffff880412525000: This client was evicted by lus13-OST000b; in progress operations using this service will fail. Mar 9 11:25:53 lustre-utils01 kernel: [109469.918673] LustreError: 7329:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff880106220af8[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x1000d0000:0x725c5f:0x0] { Mar 9 11:25:53 lustre-utils01 kernel: [109469.918906] LustreError: 7329:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999d640: [17 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:25:53 lustre-utils01 kernel: [109469.919129] LustreError: 7329:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff88010d52d170: ffff8802432ba200 0x20000041001 0x54dedc2b107ce31c 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332197 blocks: 0 Mar 9 11:25:53 lustre-utils01 kernel: [109469.919381] LustreError: 7329:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff880106220af8 Mar 9 11:25:53 lustre-utils01 kernel: [109469.919517] LustreError: 7329:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:25:53 lustre-utils01 kernel: [109469.919681] LustreError: 7329:0:(ldlm_resource.c:809:ldlm_resource_complain()) lus13-OST000d-osc-ffff880412525000: namespace resource [0x725c5f:0x0:0x0].0 (ffff880378b9cc80) refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 9 11:25:53 lustre-utils01 kernel: [109469.919935] LustreError: 7329:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x725c5f:0x0:0x0].0 (ffff880378b9cc80) refcount = 2 Mar 9 11:25:53 lustre-utils01 kernel: [109469.920101] LustreError: 7329:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:25:53 lustre-utils01 kernel: [109469.920259] LustreError: 7329:0:(ldlm_resource.c:1454:ldlm_resource_dump()) ### ### ns: lus13-OST000d-osc-ffff880412525000 lock: ffff8802432ba200/0x54dedc2b107ce31c lrc: 3/1,0 mode: PR/PR res: [0x725c5f:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x126400000000 nid: local remote: 0x23563429007686ae expref: -99 pid: 5437 timeout: 0 lvb_type: 1 Mar 9 11:25:53 lustre-utils01 kernel: [109470.262125] LustreError: 7328:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff88015c683ca8[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x1000c0000:0x72734a:0x0] { Mar 9 11:25:53 lustre-utils01 kernel: [109470.262355] LustreError: 7328:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999d240: [11 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:25:53 lustre-utils01 kernel: [109470.262567] LustreError: 7328:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff880396e22e60: ffff88023615f400 0x20000041001 0x54dedc2b107ce2f2 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332181 blocks: 0 Mar 9 11:25:53 lustre-utils01 kernel: [109470.262822] LustreError: 7328:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff88015c683ca8 Mar 9 11:25:53 lustre-utils01 kernel: [109470.262956] LustreError: 7328:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:25:53 lustre-utils01 kernel: [109470.263125] LustreError: 7328:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x72734a:0x0:0x0].0 (ffff880174053dc0) refcount = 2 Mar 9 11:25:53 lustre-utils01 kernel: [109470.263289] LustreError: 7328:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:25:53 lustre-utils01 kernel: [109470.265158] LustreError: 7327:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff88012092d948[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x1000b0000:0x7260fc:0x0] { Mar 9 11:25:53 lustre-utils01 kernel: [109470.265375] LustreError: 7327:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999d300: [5 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:25:53 lustre-utils01 kernel: [109470.265653] LustreError: 7327:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff8803745a4c38: ffff880215719800 0x20000041001 0x54dedc2b107ce2c8 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332164 blocks: 0 Mar 9 11:25:53 lustre-utils01 kernel: [109470.265898] LustreError: 7327:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff88012092d948 Mar 9 11:25:53 lustre-utils01 kernel: [109470.266038] LustreError: 7327:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:25:53 lustre-utils01 kernel: [109470.268899] LustreError: 7327:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x7260fc:0x0:0x0].0 (ffff88020f284a00) refcount = 2 Mar 9 11:25:53 lustre-utils01 kernel: [109470.269081] LustreError: 7327:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:26:03 lustre-utils01 kernel: [109480.412847] LustreError: 167-0: lus13-OST0011-osc-ffff880412525000: This client was evicted by lus13-OST0011; in progress operations using this service will fail. Mar 9 11:26:03 lustre-utils01 kernel: [109480.413094] LustreError: Skipped 3 previous similar messages Mar 9 11:26:05 lustre-utils01 kernel: [109481.956081] LustreError: 7510:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff8801799c5ca8[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100110000:0x7276e0:0x0] { Mar 9 11:26:05 lustre-utils01 kernel: [109481.956339] LustreError: 7510:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999d940: [13 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:26:05 lustre-utils01 kernel: [109481.956554] LustreError: 7510:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff880396e22da8: ffff88023615fa00 0x20000041001 0x54dedc2b107ce300 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332180 blocks: 0 Mar 9 11:26:05 lustre-utils01 kernel: [109481.956804] LustreError: 7510:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff8801799c5ca8 Mar 9 11:26:05 lustre-utils01 kernel: [109481.956934] LustreError: 7510:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:26:05 lustre-utils01 kernel: [109481.957113] LustreError: 7510:0:(ldlm_resource.c:809:ldlm_resource_complain()) lus13-OST0011-osc-ffff880412525000: namespace resource [0x7276e0:0x0:0x0].0 (ffff8801c6b04dc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 9 11:26:05 lustre-utils01 kernel: [109481.957362] LustreError: 7510:0:(ldlm_resource.c:809:ldlm_resource_complain()) Skipped 2 previous similar messages Mar 9 11:26:05 lustre-utils01 kernel: [109481.957506] LustreError: 7510:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x7276e0:0x0:0x0].0 (ffff8801c6b04dc0) refcount = 2 Mar 9 11:26:05 lustre-utils01 kernel: [109481.957670] LustreError: 7510:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:26:05 lustre-utils01 kernel: [109481.957813] LustreError: 7510:0:(ldlm_resource.c:1454:ldlm_resource_dump()) ### ### ns: lus13-OST0011-osc-ffff880412525000 lock: ffff88023615fa00/0x54dedc2b107ce300 lrc: 3/1,0 mode: PR/PR res: [0x7276e0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x126400000000 nid: local remote: 0x817cdc2fd5c94c48 expref: -99 pid: 5437 timeout: 0 lvb_type: 1 Mar 9 11:26:05 lustre-utils01 kernel: [109481.958220] LustreError: 7510:0:(ldlm_resource.c:1454:ldlm_resource_dump()) Skipped 2 previous similar messages Mar 9 11:26:05 lustre-utils01 kernel: [109481.980081] LustreError: 7511:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff880394456d80[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100120000:0x727273:0x0] { Mar 9 11:26:05 lustre-utils01 kernel: [109481.980308] LustreError: 7511:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999dec0: [7 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:26:05 lustre-utils01 kernel: [109481.980538] LustreError: 7511:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff880156b07170: ffff880215719e00 0x20000041001 0x54dedc2b107ce2d6 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332196 blocks: 0 Mar 9 11:26:05 lustre-utils01 kernel: [109481.980805] LustreError: 7511:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff880394456d80 Mar 9 11:26:05 lustre-utils01 kernel: [109481.980935] LustreError: 7511:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:26:05 lustre-utils01 kernel: [109481.981106] LustreError: 7511:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x727273:0x0:0x0].0 (ffff880178c2c500) refcount = 2 Mar 9 11:26:05 lustre-utils01 kernel: [109481.981271] LustreError: 7511:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:26:06 lustre-utils01 kernel: [109482.835758] LustreError: 11-0: lus13-OST0018-osc-ffff880412525000: Communicating with 172.17.117.148@tcp, operation obd_ping failed with -107. Mar 9 11:26:07 lustre-utils01 kernel: [109482.835948] LustreError: Skipped 7 previous similar messages Mar 9 11:26:08 lustre-utils01 kernel: [109484.939425] LustreError: 11-0: lus13-OST001b-osc-ffff880412525000: Communicating with 172.17.117.149@tcp, operation obd_ping failed with -107. Mar 9 11:26:11 lustre-utils01 kernel: [109484.939654] LustreError: Skipped 13 previous similar messages Mar 9 11:26:11 lustre-utils01 kernel: [109485.418448] LustreError: 167-0: lus13-OST001d-osc-ffff880412525000: This client was evicted by lus13-OST001d; in progress operations using this service will fail. Mar 9 11:26:11 lustre-utils01 kernel: [109485.418663] LustreError: Skipped 4 previous similar messages Mar 9 11:26:11 lustre-utils01 kernel: [109485.693234] Lustre: lus13-OST0018-osc-ffff880412525000: Connection restored to lus13-OST0018 (at 172.17.117.148@tcp) Mar 9 11:26:11 lustre-utils01 kernel: [109485.693239] Lustre: Skipped 23 previous similar messages Mar 9 11:26:11 lustre-utils01 kernel: [109487.192398] LustreError: 7594:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff8803e2bda6c0[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x1001b0000:0x72643a:0x0] { Mar 9 11:26:11 lustre-utils01 kernel: [109487.192632] LustreError: 7594:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999d8c0: [37 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:26:11 lustre-utils01 kernel: [109487.192860] LustreError: 7594:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff88028d997228: ffff8803682d9600 0x20000041001 0x54dedc2b107ce3a8 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332180 blocks: 0 Mar 9 11:26:11 lustre-utils01 kernel: [109487.193107] LustreError: 7594:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff8803e2bda6c0 Mar 9 11:26:11 lustre-utils01 kernel: [109487.193239] LustreError: 7594:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:26:11 lustre-utils01 kernel: [109487.193399] LustreError: 7594:0:(ldlm_resource.c:809:ldlm_resource_complain()) lus13-OST001b-osc-ffff880412525000: namespace resource [0x72643a:0x0:0x0].0 (ffff880229deba00) refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 9 11:26:11 lustre-utils01 kernel: [109487.193705] LustreError: 7594:0:(ldlm_resource.c:809:ldlm_resource_complain()) Skipped 1 previous similar message Mar 9 11:26:11 lustre-utils01 kernel: [109487.193862] LustreError: 7594:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x72643a:0x0:0x0].0 (ffff880229deba00) refcount = 2 Mar 9 11:26:11 lustre-utils01 kernel: [109487.194261] LustreError: 7594:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:26:11 lustre-utils01 kernel: [109487.194407] LustreError: 7594:0:(ldlm_resource.c:1454:ldlm_resource_dump()) ### ### ns: lus13-OST001b-osc-ffff880412525000 lock: ffff8803682d9600/0x54dedc2b107ce3a8 lrc: 3/1,0 mode: PR/PR res: [0x72643a:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x126400000000 nid: local remote: 0xe3112a42e0ef64c8 expref: -99 pid: 5437 timeout: 0 lvb_type: 1 Mar 9 11:26:11 lustre-utils01 kernel: [109487.194819] LustreError: 7594:0:(ldlm_resource.c:1454:ldlm_resource_dump()) Skipped 1 previous similar message Mar 9 11:26:11 lustre-utils01 kernel: [109487.745315] LustreError: 7603:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff8802d4c99d80[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100220000:0x724924:0x0] { Mar 9 11:26:11 lustre-utils01 kernel: [109487.745558] LustreError: 7603:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999dc00: [19 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:26:11 lustre-utils01 kernel: [109487.745773] LustreError: 7603:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff88017de9f508: ffff8802432ba800 0x20000041001 0x54dedc2b107ce32a 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332212 blocks: 0 Mar 9 11:26:11 lustre-utils01 kernel: [109487.746032] LustreError: 7603:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff8802d4c99d80 Mar 9 11:26:11 lustre-utils01 kernel: [109487.746164] LustreError: 7603:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:26:11 lustre-utils01 kernel: [109487.746323] LustreError: 7603:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x724924:0x0:0x0].0 (ffff880378b9c640) refcount = 2 Mar 9 11:26:11 lustre-utils01 kernel: [109487.746496] LustreError: 7603:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:26:11 lustre-utils01 kernel: [109487.793355] LustreError: 7605:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff8803e2bda438[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100200000:0x726411:0x0] { Mar 9 11:26:11 lustre-utils01 kernel: [109487.793596] LustreError: 7605:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999d740: [35 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:26:11 lustre-utils01 kernel: [109487.793814] LustreError: 7605:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff8801ca6d82e0: ffff8801a4f7bc00 0x20000041001 0x54dedc2b107ce39a 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332180 blocks: 0 Mar 9 11:26:11 lustre-utils01 kernel: [109487.794059] LustreError: 7605:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff8803e2bda438 Mar 9 11:26:11 lustre-utils01 kernel: [109487.794190] LustreError: 7605:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:26:11 lustre-utils01 kernel: [109487.794351] LustreError: 7605:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x726411:0x0:0x0].0 (ffff880229deb000) refcount = 2 Mar 9 11:26:11 lustre-utils01 kernel: [109487.794544] LustreError: 7605:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:26:11 lustre-utils01 kernel: [109487.818849] LustreError: 7604:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff880034237798[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100210000:0x7237b2:0x0] { Mar 9 11:26:11 lustre-utils01 kernel: [109487.819081] LustreError: 7604:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999d4c0: [3 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:26:11 lustre-utils01 kernel: [109487.819293] LustreError: 7604:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff8800715f7f18: ffff8801fdf32200 0x20000041001 0x54dedc2b107ce2ba 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332195 blocks: 0 Mar 9 11:26:11 lustre-utils01 kernel: [109487.819557] LustreError: 7604:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff880034237798 Mar 9 11:26:11 lustre-utils01 kernel: [109487.820678] LustreError: 7604:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:26:11 lustre-utils01 kernel: [109487.820864] LustreError: 7604:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x7237b2:0x0:0x0].0 (ffff8803e2a8c140) refcount = 2 Mar 9 11:26:11 lustre-utils01 kernel: [109487.821030] LustreError: 7604:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:26:12 lustre-utils01 kernel: [109488.930623] LustreError: 7613:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lock@ffff8803e2bda000[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100260000:0x728268:0x0] { Mar 9 11:26:12 lustre-utils01 kernel: [109488.930860] LustreError: 7613:0:(osc_lock.c:834:osc_ldlm_completion_ast()) lovsub@ffff88016999d880: [31 ffff8802e69c72d8 P(0):[0, 18446744073709551615]@[0x2000009b4:0xb4e7:0x0]] Mar 9 11:26:12 lustre-utils01 kernel: [109488.931088] LustreError: 7613:0:(osc_lock.c:834:osc_ldlm_completion_ast()) osc@ffff88034d400450: ffff8801a4f7ba00 0x20000041001 0x54dedc2b107ce37e 3 ffff880174f9d238 size: 0 mtime: 0 atime: 0 ctime: 1367332196 blocks: 0 Mar 9 11:26:12 lustre-utils01 kernel: [109488.931330] LustreError: 7613:0:(osc_lock.c:834:osc_ldlm_completion_ast()) } lock@ffff8803e2bda000 Mar 9 11:26:12 lustre-utils01 kernel: [109488.931460] LustreError: 7613:0:(osc_lock.c:834:osc_ldlm_completion_ast()) dlmlock returned -5 Mar 9 11:26:12 lustre-utils01 kernel: [109488.931617] LustreError: 7613:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0x728268:0x0:0x0].0 (ffff88023351db40) refcount = 2 Mar 9 11:26:12 lustre-utils01 kernel: [109488.931781] LustreError: 7613:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): Mar 9 11:28:04 lustre-utils01 kernel: [109600.859319] LustreError: 22493:0:(cl_lock.c:1414:cl_unuse_try()) result = -5, this is unlikely! Mar 9 13:13:19 lustre-utils01 kernel: [115912.915780] warn_alloc_failed: 31 callbacks suppressed Mar 9 13:13:55 lustre-utils01 kernel: [115912.915793] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:55 lustre-utils01 kernel: [115912.915799] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:55 lustre-utils01 kernel: [115912.915800] Call Trace: Mar 9 13:13:55 lustre-utils01 kernel: [115912.915836] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:55 lustre-utils01 kernel: [115912.915840] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:55 lustre-utils01 kernel: [115912.915857] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:55 lustre-utils01 kernel: [115912.915865] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:55 lustre-utils01 kernel: [115912.915867] [] new_slab+0x30/0x1a0 Mar 9 13:13:55 lustre-utils01 kernel: [115912.915879] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:55 lustre-utils01 kernel: [115912.915902] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 13:13:55 lustre-utils01 kernel: [115912.915909] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 13:13:55 lustre-utils01 kernel: [115912.915911] [] kmem_cache_alloc_trace+0x132/0x140 Mar 9 13:13:55 lustre-utils01 kernel: [115912.915917] [] ? lnet_md_alloc.isra.2+0x74/0x280 [lnet] Mar 9 13:13:55 lustre-utils01 kernel: [115912.915923] [] LNetPut+0x122/0x900 [lnet] Mar 9 13:13:55 lustre-utils01 kernel: [115912.915946] [] ? cfs_percpt_unlock+0x42/0xd0 [libcfs] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916002] [] ptl_send_buf+0x12f/0x5d0 [ptlrpc] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916009] [] ? LNetMDAttach+0x2db/0x370 [lnet] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916016] [] ? read_tsc+0x9/0x20 Mar 9 13:13:55 lustre-utils01 kernel: [115912.916037] [] ptl_send_rpc+0x522/0xc20 [ptlrpc] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916057] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916076] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916096] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916123] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916145] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916158] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:55 lustre-utils01 kernel: [115912.916180] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:55 lustre-utils01 kernel: [115912.916190] [] kthread+0xc0/0xd0 Mar 9 13:13:55 lustre-utils01 kernel: [115912.916193] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:55 lustre-utils01 kernel: [115912.916203] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:55 lustre-utils01 kernel: [115912.916206] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:55 lustre-utils01 kernel: [115912.916207] Mem-Info: Mar 9 13:13:55 lustre-utils01 kernel: [115912.916209] Node 0 DMA per-cpu: Mar 9 13:13:55 lustre-utils01 kernel: [115912.916211] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:55 lustre-utils01 kernel: [115912.916212] Node 0 DMA32 per-cpu: Mar 9 13:13:55 lustre-utils01 kernel: [115912.916213] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:55 lustre-utils01 kernel: [115912.916214] Node 0 Normal per-cpu: Mar 9 13:13:55 lustre-utils01 kernel: [115912.916215] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916219] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916219] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916219] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916219] free:29332 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916219] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916219] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916221] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115912.916226] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916228] Node 0 DMA32 free:60264kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115912.916232] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916234] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115912.916237] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916239] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.916247] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1336*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60232kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.916259] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.916265] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115912.916267] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115912.916268] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115912.916269] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.916270] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.964783] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115912.964787] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115912.964788] 314825 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115912.964788] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115912.964793] __slab_alloc: 31 callbacks suppressed Mar 9 13:13:56 lustre-utils01 kernel: [115912.964795] SLUB: Unable to allocate memory on node -1 (gfp=0x40) Mar 9 13:13:56 lustre-utils01 kernel: [115912.964796] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.964798] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.964955] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: out of memory at /var/lib/dkms/lustre-client-modules-2.5.0/2.5.0/build/lnet/include/lnet/lib-lnet.h:506 (tried to alloc '(msg)' = 352) Mar 9 13:13:56 lustre-utils01 kernel: [115912.965241] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) Skipped 9 previous similar messages Mar 9 13:13:56 lustre-utils01 kernel: [115912.965388] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: 725950087 total bytes allocated by lnet Mar 9 13:13:56 lustre-utils01 kernel: [115912.965528] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) Skipped 9 previous similar messages Mar 9 13:13:56 lustre-utils01 kernel: [115912.965687] LNetError: 2001:0:(lib-move.c:2138:LNetPut()) Dropping PUT to 12345-172.17.117.151@tcp: ENOMEM on lnet_msg_t Mar 9 13:13:56 lustre-utils01 kernel: [115912.965842] LNetError: 2001:0:(lib-move.c:2138:LNetPut()) Skipped 9 previous similar messages Mar 9 13:13:56 lustre-utils01 kernel: [115912.965978] LustreError: 2001:0:(niobuf.c:96:ptl_send_buf()) LNetPut(12345-172.17.117.151@tcp, 17, 1461982928011904) failed: -12 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966140] LustreError: 2001:0:(niobuf.c:96:ptl_send_buf()) Skipped 9 previous similar messages Mar 9 13:13:56 lustre-utils01 kernel: [115912.966305] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1394370799/real 1394370799] req@ffff880420856000 x1461982928011904/t0(0) o103->lus13-OST0024-osc-ffff880412525000@172.17.117.151@tcp:17/18 lens 344/224 e 0 to 1 dl 1394371029 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966307] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Mar 9 13:13:56 lustre-utils01 kernel: [115912.966317] Lustre: lus13-OST0024-osc-ffff880412525000: Connection to lus13-OST0024 (at 172.17.117.151@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 13:13:56 lustre-utils01 kernel: [115912.966319] Lustre: Skipped 2 previous similar messages Mar 9 13:13:56 lustre-utils01 kernel: [115912.966342] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966345] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115912.966346] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115912.966357] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966360] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966376] [] ? console_unlock.part.8+0x22e/0x280 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966381] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966388] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966390] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966394] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966438] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966443] [] ? check_preempt_curr+0x84/0xa0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966445] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966447] [] ? ttwu_do_activate.constprop.86+0x66/0x70 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966469] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966491] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966511] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966532] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966534] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966558] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966580] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966600] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966620] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966641] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966662] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966665] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966686] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115912.966689] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966691] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966694] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966696] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966699] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115912.966700] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115912.966703] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966705] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115912.966706] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966707] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115912.966709] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966713] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966713] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966713] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966713] free:29332 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966713] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966713] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966717] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115912.966723] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966726] Node 0 DMA32 free:60264kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115912.966732] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966734] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115912.966738] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966740] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.966747] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1336*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60232kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.966755] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.966761] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115912.966762] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115912.966763] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115912.966764] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115912.966765] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.014342] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.014346] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.014347] 314825 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.014347] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.014353] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.014355] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014388] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014476] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014478] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115913.014480] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115913.014491] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014504] [] ? number.isra.2+0x355/0x390 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014506] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014509] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014510] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014533] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014537] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014542] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014545] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014549] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014598] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014607] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014611] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014636] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014659] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014679] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014704] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014712] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014740] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014765] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014788] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014811] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014833] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014855] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014858] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014879] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.014882] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014884] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014888] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014890] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014891] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115913.014893] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.014895] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014895] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.014897] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014898] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.014899] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014902] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014902] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014902] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014902] free:29332 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014902] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014902] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014905] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115913.014909] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014912] Node 0 DMA32 free:60264kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.014916] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014917] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.014933] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014936] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.014943] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1336*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60232kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.014951] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.014957] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115913.014958] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115913.014960] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115913.014961] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.014961] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.065754] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.065760] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.065761] 314825 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.065761] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.065768] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.065771] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065773] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065891] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065894] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115913.065896] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115913.065907] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065911] [] ? number.isra.2+0x355/0x390 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065913] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065916] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065917] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065940] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.065944] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065949] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065952] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.065957] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066012] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066023] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066027] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066055] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066084] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066112] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066137] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066146] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066168] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066196] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066226] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066249] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066272] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066293] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066296] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066317] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.066321] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066323] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066326] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066328] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066329] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115913.066331] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.066333] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066334] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.066349] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066350] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.066352] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066355] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066355] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066355] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066355] free:29332 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066355] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066355] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066358] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115913.066362] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066364] Node 0 DMA32 free:60264kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.066368] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066370] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.066373] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066375] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.066383] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1336*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60232kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.066390] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.066396] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115913.066398] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115913.066399] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115913.066400] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.066401] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.118555] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.118559] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.118560] 314825 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.118561] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.118566] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.118568] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118570] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118733] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118736] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115913.118741] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115913.118755] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118760] [] ? number.isra.2+0x355/0x390 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118762] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118765] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118767] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118789] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.118792] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118795] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118797] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118801] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118858] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.118866] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.118869] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118891] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.118915] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.118938] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.118959] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.118963] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115913.118986] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.119009] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.119029] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.119048] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.119069] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.119090] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.119093] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119114] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.119118] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119120] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119123] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119125] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119127] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115913.119128] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.119131] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119131] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.119133] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119134] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.119135] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119138] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119138] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119138] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119138] free:29332 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119138] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119138] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119141] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115913.119145] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119148] Node 0 DMA32 free:60264kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.119152] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119154] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.119157] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119159] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.119167] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1336*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60232kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.119174] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.119187] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115913.119189] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115913.119190] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115913.119191] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.119192] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.171081] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.171085] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.171086] 314825 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.171086] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.171092] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.171094] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171096] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171290] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171293] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115913.171295] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115913.171305] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171310] [] ? number.isra.2+0x355/0x390 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171312] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171314] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171316] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171338] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171342] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171344] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171346] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171349] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171395] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171402] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171404] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171426] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171448] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171467] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171490] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171498] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171523] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171552] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171578] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171603] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171632] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171662] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171667] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171690] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.171695] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171697] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171702] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171704] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171705] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115913.171706] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.171708] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171709] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.171711] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171712] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.171713] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171716] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171716] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171716] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171716] free:29316 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171716] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171716] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171719] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115913.171723] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171743] Node 0 DMA32 free:60200kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.171747] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171749] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.171753] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171755] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.171763] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1335*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60200kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.171770] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.171776] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115913.171777] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115913.171779] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115913.171780] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.171780] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.222527] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.222531] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.222532] 314929 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.222533] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.222538] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.222540] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222542] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222644] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222647] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115913.222649] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115913.222660] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222664] [] ? number.isra.2+0x355/0x390 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222666] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222668] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222670] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222692] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222696] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222699] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222701] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222704] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222748] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222755] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222758] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222779] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222801] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222820] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222841] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222845] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115913.222865] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222886] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222906] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222925] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222946] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222991] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.222995] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223016] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.223020] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223022] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223025] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223027] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223029] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115913.223030] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.223032] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223033] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.223034] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223035] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.223037] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223040] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223040] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223040] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223040] free:29316 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223040] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223040] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223043] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115913.223047] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223049] Node 0 DMA32 free:60200kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.223053] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223055] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.223059] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223060] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.223068] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1335*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60200kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.223075] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.223081] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115913.223083] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115913.223084] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115913.223085] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.223086] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.270483] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.270488] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.270489] 314929 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.270489] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.270494] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.270496] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270498] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270596] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270599] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115913.270601] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115913.270612] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270636] [] ? number.isra.2+0x355/0x390 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270638] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270641] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270643] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270665] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270669] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270672] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270673] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270677] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270722] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270729] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270731] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270753] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270775] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270794] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270816] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270820] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270840] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270861] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270881] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270900] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270922] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270943] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270945] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270966] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.270970] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270972] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270975] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270977] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270979] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115913.270980] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.270982] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270983] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.270984] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270985] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.270987] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270990] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270990] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270990] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270990] free:29316 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270990] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270990] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.270993] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115913.270998] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115913.271000] Node 0 DMA32 free:60200kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.271004] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115913.271006] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.271016] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.271018] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.271026] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1335*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60200kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.271034] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.271040] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115913.271041] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115913.271043] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115913.271044] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.271044] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.317960] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.317964] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.317964] 314929 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.317965] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.317969] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.317972] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.317974] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318061] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318064] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115913.318065] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115913.318076] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318081] [] ? number.isra.2+0x355/0x390 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318082] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318085] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318087] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318108] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318112] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318115] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318117] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318120] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318163] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318170] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318172] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318194] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318215] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318235] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318256] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318260] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318280] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318301] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318321] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318340] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318363] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318385] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318388] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318409] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.318413] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318415] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318418] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318420] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318422] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115913.318423] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.318425] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318426] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.318427] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318428] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.318429] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318433] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318433] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318433] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318433] free:29316 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318433] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318433] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318435] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115913.318440] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318442] Node 0 DMA32 free:60200kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.318446] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318448] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.318451] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318453] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.318461] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1335*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60200kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.318468] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.318474] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115913.318476] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115913.318477] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115913.318478] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.318478] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.365568] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.365571] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.365572] 314929 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.365573] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.365578] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.365580] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365582] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365740] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365743] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 13:13:56 lustre-utils01 kernel: [115913.365745] Call Trace: Mar 9 13:13:56 lustre-utils01 kernel: [115913.365755] [] warn_alloc_failed+0xf6/0x150 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365760] [] ? number.isra.2+0x355/0x390 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365761] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365764] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365766] [] ? put_dec+0x72/0x90 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365788] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.365792] [] alloc_pages_current+0xba/0x170 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365794] [] allocate_slab+0x190/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365796] [] new_slab+0x30/0x1a0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365800] [] __slab_alloc+0x17d/0x2b2 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365844] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.365851] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 13:13:56 lustre-utils01 kernel: [115913.365853] [] __kmalloc+0x162/0x190 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365877] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.365899] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.365919] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.365940] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.365944] [] ? default_wake_function+0x12/0x20 Mar 9 13:13:56 lustre-utils01 kernel: [115913.365963] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.365984] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.366004] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.366023] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.366045] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.366066] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.366068] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366090] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:13:56 lustre-utils01 kernel: [115913.366093] [] kthread+0xc0/0xd0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366095] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366098] [] ret_from_fork+0x7c/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366101] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366102] Mem-Info: Mar 9 13:13:56 lustre-utils01 kernel: [115913.366103] Node 0 DMA per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.366105] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366106] Node 0 DMA32 per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.366107] CPU 0: hi: 186, btch: 31 usd: 3 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366108] Node 0 Normal per-cpu: Mar 9 13:13:56 lustre-utils01 kernel: [115913.366110] CPU 0: hi: 186, btch: 31 usd: 94 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366113] active_anon:7745 inactive_anon:27966 isolated_anon:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366113] active_file:40027 inactive_file:27752 isolated_file:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366113] unevictable:8798 dirty:0 writeback:0 unstable:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366113] free:29316 slab_reclaimable:30588 slab_unreclaimable:3731816 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366113] mapped:8851 shmem:33 pagetables:1349 bounce:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366113] free_cma:0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366116] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 13:13:56 lustre-utils01 kernel: [115913.366120] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366122] Node 0 DMA32 free:60200kB min:12604kB low:15752kB high:18904kB active_anon:4584kB inactive_anon:4676kB active_file:33408kB inactive_file:15688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:13064kB slab_unreclaimable:2917080kB kernel_stack:48kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.366126] lowmem_reserve[]: 0 0 13104 13104 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366128] Node 0 Normal free:41160kB min:54912kB low:68640kB high:82368kB active_anon:26396kB inactive_anon:107188kB active_file:126700kB inactive_file:95320kB unevictable:35192kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:35192kB dirty:0kB writeback:0kB mapped:35356kB shmem:132kB slab_reclaimable:109288kB slab_unreclaimable:12010180kB kernel_stack:2224kB pagetables:5380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 13:13:56 lustre-utils01 kernel: [115913.366131] lowmem_reserve[]: 0 0 0 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366133] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.366141] Node 0 DMA32: 120*4kB (UM) 109*8kB (UEM) 356*16kB (UEM) 1335*32kB (UEM) 65*64kB (UEM) 17*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60200kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.366148] Node 0 Normal: 9836*4kB (UM) 185*8kB (EM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41160kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.366154] 70150 total pagecache pages Mar 9 13:13:56 lustre-utils01 kernel: [115913.366156] 1492 pages in swap cache Mar 9 13:13:56 lustre-utils01 kernel: [115913.366157] Swap cache stats: add 17830, delete 16338, find 11795/12046 Mar 9 13:13:56 lustre-utils01 kernel: [115913.366158] Free swap = 38730404kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.366159] Total swap = 38793204kB Mar 9 13:13:56 lustre-utils01 kernel: [115913.413603] 4194288 pages RAM Mar 9 13:13:56 lustre-utils01 kernel: [115913.413607] 85481 pages reserved Mar 9 13:13:56 lustre-utils01 kernel: [115913.413608] 314929 pages shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.413609] 4045011 pages non-shared Mar 9 13:13:56 lustre-utils01 kernel: [115913.413614] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 13:13:56 lustre-utils01 kernel: [115913.413616] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.413618] node 0: slabs: 42495, objs: 339960, free: 0 Mar 9 13:13:56 lustre-utils01 kernel: [115913.431082] LNetError: 1999:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: out of memory at /var/lib/dkms/lustre-client-modules-2.5.0/2.5.0/build/lnet/include/lnet/lib-lnet.h:506 (tried to alloc '(msg)' = 352) Mar 9 13:13:56 lustre-utils01 kernel: [115913.431314] LNetError: 1999:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: 725950087 total bytes allocated by lnet Mar 9 13:13:56 lustre-utils01 kernel: [115913.431450] LNetError: 1999:0:(lib-move.c:2138:LNetPut()) Dropping PUT to 12345-172.17.117.153@tcp: ENOMEM on lnet_msg_t Mar 9 13:13:56 lustre-utils01 kernel: [115913.431606] LustreError: 1999:0:(niobuf.c:96:ptl_send_buf()) LNetPut(12345-172.17.117.153@tcp, 12, 1461982928021296) failed: -12 Mar 9 13:16:13 lustre-utils01 kernel: [116085.952632] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394370824/real 1394370824] req@ffff8802b6d42000 x1461982928596244/t0(0) o38->lus13-MDT0000-mdc-ffff880412525000@172.17.117.152@tcp:12/10 lens 400/544 e 0 to 1 dl 1394370929 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 13:16:13 lustre-utils01 kernel: [116085.952637] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 79 previous similar messages Mar 9 13:16:13 lustre-utils01 kernel: [116085.952707] Lustre: lus13-OST0000-osc-ffff880412525000: Connection restored to lus13-OST0000 (at 172.17.117.144@tcp) Mar 9 13:16:13 lustre-utils01 kernel: [116085.952708] Lustre: Skipped 15 previous similar messages Mar 9 13:16:23 lustre-utils01 kernel: [116096.476834] Lustre: lus13-OST0002-osc-ffff880412525000: Connection restored to lus13-OST0002 (at 172.17.117.144@tcp) Mar 9 13:16:29 lustre-utils01 kernel: [116096.476839] Lustre: Skipped 1 previous similar message Mar 9 13:16:38 lustre-utils01 kernel: [116111.648573] INFO: task ldlm_bl_01:2011 blocked for more than 120 seconds. Mar 9 13:16:38 lustre-utils01 kernel: [116111.648702] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 9 13:16:38 lustre-utils01 kernel: [116111.648824] ldlm_bl_01 D ffff880425a2be60 0 2011 2 0x00000000 Mar 9 13:16:38 lustre-utils01 kernel: [116111.648829] ffff88042144ba98 0000000000000046 ffffffffa0564ba3 ffff88043fc13ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.648832] ffff88042144bfd8 ffff88042144bfd8 ffff88042144bfd8 0000000000013ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.648834] ffff88035e040000 ffff880427a35d00 ffff88042144ba78 ffff880428cb7d00 Mar 9 13:16:38 lustre-utils01 kernel: [116111.648836] Call Trace: Mar 9 13:16:38 lustre-utils01 kernel: [116111.648855] [] schedule+0x29/0x70 Mar 9 13:16:38 lustre-utils01 kernel: [116111.648858] [] schedule_preempt_disabled+0xe/0x10 Mar 9 13:16:38 lustre-utils01 kernel: [116111.648861] [] __mutex_lock_slowpath+0xd7/0x150 Mar 9 13:16:38 lustre-utils01 kernel: [116111.648863] [] mutex_lock+0x2a/0x50 Mar 9 13:16:38 lustre-utils01 kernel: [116111.648886] [] mdc_close+0x1bf/0xa90 [mdc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.648900] [] lmv_close+0x26f/0x5b0 [lmv] Mar 9 13:16:38 lustre-utils01 kernel: [116111.648935] [] ll_close_inode_openhandle+0x33f/0xff0 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.648947] [] ll_md_real_close+0x151/0x220 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.648973] [] ll_md_blocking_ast+0x2f9/0x790 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649015] [] ? ptlrpc_set_add_new_req+0x91/0x180 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649040] [] ? ptlrpcd_add_req+0x1b1/0x330 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649056] [] ldlm_cancel_callback+0x6f/0x160 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649073] [] ldlm_cli_cancel_local+0x97/0x440 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649090] [] ldlm_cli_cancel_list_local+0xe5/0x290 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649108] [] ldlm_bl_thread_main+0xca/0x460 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649112] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649130] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649133] [] kthread+0xc0/0xd0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649136] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649139] [] ret_from_fork+0x7c/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649141] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649158] INFO: task ldlm_bl_19:23889 blocked for more than 120 seconds. Mar 9 13:16:38 lustre-utils01 kernel: [116111.649255] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 9 13:16:38 lustre-utils01 kernel: [116111.649373] ldlm_bl_19 D ffff880425a2be60 0 23889 2 0x00000000 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649375] ffff8803abd11a98 0000000000000046 ffffffffa0564ba3 ffff88043fc13ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649377] ffff8803abd11fd8 ffff8803abd11fd8 ffff8803abd11fd8 0000000000013ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649379] ffff880427e98000 ffff8803d5e45d00 ffff8803abd11a78 ffff880428cb7d00 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649381] Call Trace: Mar 9 13:16:38 lustre-utils01 kernel: [116111.649384] [] schedule+0x29/0x70 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649386] [] schedule_preempt_disabled+0xe/0x10 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649388] [] __mutex_lock_slowpath+0xd7/0x150 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649390] [] mutex_lock+0x2a/0x50 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649394] [] mdc_close+0x1bf/0xa90 [mdc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649399] [] lmv_close+0x26f/0x5b0 [lmv] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649410] [] ll_close_inode_openhandle+0x33f/0xff0 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649421] [] ll_md_real_close+0x151/0x220 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649433] [] ll_md_blocking_ast+0x2f9/0x790 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649453] [] ? ptlrpc_set_add_new_req+0x91/0x180 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649474] [] ? ptlrpcd_add_req+0x1b1/0x330 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649490] [] ldlm_cancel_callback+0x6f/0x160 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649507] [] ldlm_cli_cancel_local+0x97/0x440 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649525] [] ldlm_cli_cancel_list_local+0xe5/0x290 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649543] [] ldlm_bl_thread_main+0xca/0x460 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649545] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649563] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649565] [] kthread+0xc0/0xd0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649567] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649570] [] ret_from_fork+0x7c/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649572] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649575] INFO: task ldlm_bl_25:24595 blocked for more than 120 seconds. Mar 9 13:16:38 lustre-utils01 kernel: [116111.649684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 9 13:16:38 lustre-utils01 kernel: [116111.649802] ldlm_bl_25 D ffff880425a2be60 0 24595 2 0x00000000 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649804] ffff88037cb27a98 0000000000000046 ffffffffa0564ba3 ffff88043fc13ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649806] ffff88037cb27fd8 ffff88037cb27fd8 ffff88037cb27fd8 0000000000013ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649808] ffff880427e98000 ffff8803d5f1dd00 ffff88037cb27a78 ffff880428cb7d00 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649810] Call Trace: Mar 9 13:16:38 lustre-utils01 kernel: [116111.649814] [] schedule+0x29/0x70 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649816] [] schedule_preempt_disabled+0xe/0x10 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649818] [] __mutex_lock_slowpath+0xd7/0x150 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649820] [] mutex_lock+0x2a/0x50 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649824] [] mdc_close+0x1bf/0xa90 [mdc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649829] [] lmv_close+0x26f/0x5b0 [lmv] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649840] [] ll_close_inode_openhandle+0x33f/0xff0 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649851] [] ll_md_real_close+0x151/0x220 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649863] [] ll_md_blocking_ast+0x2f9/0x790 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649883] [] ? ptlrpc_set_add_new_req+0x91/0x180 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649904] [] ? ptlrpcd_add_req+0x1b1/0x330 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649920] [] ldlm_cancel_callback+0x6f/0x160 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649937] [] ldlm_cli_cancel_local+0x97/0x440 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649954] [] ldlm_cli_cancel_list_local+0xe5/0x290 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649972] [] ldlm_bl_thread_main+0xca/0x460 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649975] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649992] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.649995] [] kthread+0xc0/0xd0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649997] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.649999] [] ret_from_fork+0x7c/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650001] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650006] INFO: task ldlm_bl_48:26414 blocked for more than 120 seconds. Mar 9 13:16:38 lustre-utils01 kernel: [116111.650102] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 9 13:16:38 lustre-utils01 kernel: [116111.650220] ldlm_bl_48 D ffff880425a2be60 0 26414 2 0x00000000 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650229] ffff880191cc3a98 0000000000000046 ffffffffa0564ba3 ffff88043fc13ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650231] ffff880191cc3fd8 ffff880191cc3fd8 ffff880191cc3fd8 0000000000013ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650233] ffff880320079740 ffff8802fa418000 ffff880191cc3a78 ffff880428cb7d00 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650235] Call Trace: Mar 9 13:16:38 lustre-utils01 kernel: [116111.650238] [] schedule+0x29/0x70 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650240] [] schedule_preempt_disabled+0xe/0x10 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650242] [] __mutex_lock_slowpath+0xd7/0x150 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650244] [] mutex_lock+0x2a/0x50 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650248] [] mdc_close+0x1bf/0xa90 [mdc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650251] [] ? set_next_entity+0xa5/0xc0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650256] [] lmv_close+0x26f/0x5b0 [lmv] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650266] [] ll_close_inode_openhandle+0x33f/0xff0 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650277] [] ll_md_real_close+0x151/0x220 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650290] [] ll_md_blocking_ast+0x2f9/0x790 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650309] [] ? ptlrpc_set_add_new_req+0x91/0x180 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650331] [] ? ptlrpcd_add_req+0x1b1/0x330 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650347] [] ldlm_cancel_callback+0x6f/0x160 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650364] [] ldlm_cli_cancel_local+0x97/0x440 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650381] [] ldlm_cli_cancel_list_local+0xe5/0x290 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650399] [] ldlm_bl_thread_main+0xca/0x460 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650401] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650419] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650421] [] kthread+0xc0/0xd0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650423] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650426] [] ret_from_fork+0x7c/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650428] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650429] INFO: task ldlm_bl_48:26415 blocked for more than 120 seconds. Mar 9 13:16:38 lustre-utils01 kernel: [116111.650526] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 9 13:16:38 lustre-utils01 kernel: [116111.650654] ldlm_bl_48 D ffff880425a2be60 0 26415 2 0x00000000 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650657] ffff8801a3469a98 0000000000000046 ffffffffa0564ba3 ffff88043fc13ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650659] ffff8801a3469fd8 ffff8801a3469fd8 ffff8801a3469fd8 0000000000013ec0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650661] ffff880320079740 ffff8802fa419740 ffff8801a3469a78 ffff880428cb7d00 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650662] Call Trace: Mar 9 13:16:38 lustre-utils01 kernel: [116111.650666] [] schedule+0x29/0x70 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650668] [] schedule_preempt_disabled+0xe/0x10 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650670] [] __mutex_lock_slowpath+0xd7/0x150 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650672] [] mutex_lock+0x2a/0x50 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650677] [] mdc_close+0x1bf/0xa90 [mdc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650682] [] lmv_close+0x26f/0x5b0 [lmv] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650693] [] ll_close_inode_openhandle+0x33f/0xff0 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650703] [] ll_md_real_close+0x151/0x220 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650716] [] ll_md_blocking_ast+0x2f9/0x790 [lustre] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650735] [] ? ptlrpc_set_add_new_req+0x91/0x180 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650757] [] ? ptlrpcd_add_req+0x1b1/0x330 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650772] [] ldlm_cancel_callback+0x6f/0x160 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650789] [] ldlm_cli_cancel_local+0x97/0x440 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650806] [] ldlm_cli_cancel_list_local+0xe5/0x290 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650824] [] ldlm_bl_thread_main+0xca/0x460 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650826] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650844] [] ? ldlm_handle_bl_callback+0x3e0/0x3e0 [ptlrpc] Mar 9 13:16:38 lustre-utils01 kernel: [116111.650846] [] kthread+0xc0/0xd0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650849] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650851] [] ret_from_fork+0x7c/0xb0 Mar 9 13:16:38 lustre-utils01 kernel: [116111.650853] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:16:39 lustre-utils01 kernel: [116112.914903] Lustre: lus13-OST0007-osc-ffff880412525000: Connection restored to lus13-OST0007 (at 172.17.117.145@tcp) Mar 9 13:16:40 lustre-utils01 kernel: [116112.914926] Lustre: Skipped 4 previous similar messages Mar 9 13:17:11 lustre-utils01 kernel: [116144.677014] LustreError: 167-0: lus13-MDT0000-mdc-ffff880412525000: This client was evicted by lus13-MDT0000; in progress operations using this service will fail. Mar 9 13:17:11 lustre-utils01 kernel: [116144.677233] LustreError: Skipped 10 previous similar messages Mar 9 13:17:12 lustre-utils01 kernel: [116144.686511] LustreError: 26410:0:(file.c:171:ll_close_inode_openhandle()) inode 144115373078263769 mdc close failed: rc = -5 Mar 9 13:17:13 lustre-utils01 kernel: [116146.585391] LustreError: 2001:0:(mdc_locks.c:1254:mdc_intent_getattr_async_interpret()) ldlm_cli_enqueue_fini: -5 Mar 9 13:17:13 lustre-utils01 kernel: [116147.287126] LustreError: 9986:0:(mdc_locks.c:915:mdc_enqueue()) ldlm_cli_enqueue: -5 Mar 9 13:17:13 lustre-utils01 kernel: [116147.287319] LustreError: 9986:0:(file.c:3069:ll_inode_revalidate_fini()) lus13: revalidate FID [0x200002aef:0x1b0:0x0] error: rc = -5 Mar 9 13:17:13 lustre-utils01 kernel: [116147.288045] Lustre: lus13-MDT0000-mdc-ffff880412525000: Connection restored to lus13-MDT0000 (at 172.17.117.153@tcp) Mar 9 13:17:13 lustre-utils01 kernel: [116147.288047] Lustre: Skipped 32 previous similar messages Mar 9 13:20:08 lustre-utils01 kernel: [116320.712256] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394370977/real 0] req@ffff88013e97ac00 x1461982932837704/t0(0) o103->lus13-OST0000-osc-ffff880412525000@172.17.117.144@tcp:17/18 lens 344/224 e 0 to 1 dl 1394371207 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 13:20:08 lustre-utils01 kernel: [116320.712500] Lustre: lus13-OST0000-osc-ffff880412525000: Connection to lus13-OST0000 (at 172.17.117.144@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 13:20:08 lustre-utils01 kernel: [116320.712503] Lustre: Skipped 40 previous similar messages Mar 9 13:20:24 lustre-utils01 kernel: [116328.564026] Lustre: lus13-OST0001-osc-ffff880412525000: Connection to lus13-OST0001 (at 172.17.117.144@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 13:20:32 lustre-utils01 kernel: [116339.567622] Lustre: lus13-OST0004-osc-ffff880412525000: Connection to lus13-OST0004 (at 172.17.117.144@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 13:20:32 lustre-utils01 kernel: [116339.567627] Lustre: Skipped 2 previous similar messages Mar 9 13:20:49 lustre-utils01 kernel: [116352.697942] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394371001/real 0] req@ffff8801e32ae800 x1461982931473108/t0(0) o103->lus13-OST0004-osc-ffff880412525000@172.17.117.144@tcp:17/18 lens 344/224 e 0 to 1 dl 1394371231 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 13:20:49 lustre-utils01 kernel: [116352.697947] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 188428 previous similar messages Mar 9 13:21:17 lustre-utils01 kernel: [116364.608792] Lustre: lus13-OST000e-osc-ffff880412525000: Connection to lus13-OST000e (at 172.17.117.146@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 13:21:17 lustre-utils01 kernel: [116364.608797] Lustre: Skipped 7 previous similar messages Mar 9 13:21:17 lustre-utils01 kernel: [116385.382435] BUG: soft lockup - CPU#0 stuck for 21s! [ptlrpcd_1:2001] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382558] Modules linked in: lmv(OF) fld(OF) mgc(OF) lustre(OF) lov(OF) osc(OF) mdc(OF) fid(OF) ksocklnd(OF) ptlrpc(OF) obdclass(OF) lnet(OF) lvfs(OF) libcfs(OF) 8021q(F) garp(F) bridge(F) stp(F) llc(F) acpiphp(F) autofs4(F) binfmt_misc(F) nfsd(F) auth_rpcgss(F) ext2(F) vmwgfx(F) ttm(F) ppdev(F) coretemp(F) nfsv3(F) nfs_acl(F) nfs(F) fscache(F) lockd(F) microcode(F) sunrpc(F) vmw_balloon(F) drm(F) i2c_piix4(F) shpchp(F) psmouse(F) serio_raw(F) mac_hid(F) parport_pc(F) lp(F) parport(F) floppy(F) vmw_pvscsi(F) vmxnet3(F) libcrc32c(F) [last unloaded: ipmi_msghandler] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382630] CPU 0 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382640] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform Mar 9 13:21:17 lustre-utils01 kernel: [116385.382641] RIP: 0010:[] [] lnet_res_lh_lookup+0x56/0x70 [lnet] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382658] RSP: 0000:ffff8804270ddc08 EFLAGS: 00000206 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382659] RAX: ffff8801afb57790 RBX: ffff880400000158 RCX: ffff8803a9597e10 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382660] RDX: ffffc90012550bb0 RSI: 0000000059d582ed RDI: ffff880427a94b40 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382661] RBP: ffff8804270ddc08 R08: 0000000000000001 R09: 0000000000000000 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382662] R10: 0000000000000000 R11: 000000000000000f R12: 0000000000000011 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382669] R13: ffffffffa02d3e80 R14: ffff88041214b88c R15: ffff880400000067 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382671] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382672] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 9 13:21:17 lustre-utils01 kernel: [116385.382673] CR2: 000000000231f010 CR3: 0000000426aa3000 CR4: 00000000000007f0 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382710] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382726] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382728] Process ptlrpcd_1 (pid: 2001, threadinfo ffff8804270dc000, task ffff880427e9dd00) Mar 9 13:21:17 lustre-utils01 kernel: [116385.382729] Stack: Mar 9 13:21:17 lustre-utils01 kernel: [116385.382730] ffff8804270ddcc8 ffffffffa0305ce5 ffff880400000000 00000000ffffffff Mar 9 13:21:17 lustre-utils01 kernel: [116385.382733] 0000000000000000 00000000ffffffff 0000000000000000 ffffffffa0564ba3 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382735] ffffffffa0564ba3 ffffffffa0564ba3 ffffffffa0564ba3 ffffffffa0564ba3 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382737] Call Trace: Mar 9 13:21:17 lustre-utils01 kernel: [116385.382746] [] LNetMDUnlink+0xa5/0x1b0 [lnet] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382787] [] ptlrpc_unregister_reply+0x15c/0xa30 [ptlrpc] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382807] [] ptlrpc_expire_one_request+0xd4/0x4c0 [ptlrpc] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382827] [] ptlrpc_expired_set+0xa3/0x190 [ptlrpc] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382850] [] ptlrpcd+0x219/0x3a0 [ptlrpc] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382856] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382877] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:21:17 lustre-utils01 kernel: [116385.382881] [] kthread+0xc0/0xd0 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382883] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382887] [] ret_from_fork+0x7c/0xb0 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382889] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:21:17 lustre-utils01 kernel: [116385.382890] Code: 00 00 48 c1 e2 04 48 03 57 20 48 8b 0a 48 39 d1 74 23 48 39 71 10 48 89 c8 75 10 eb 18 0f 1f 84 00 00 00 00 00 48 39 70 10 74 0a <48> 8b 00 48 39 d0 75 f2 31 c0 5d c3 66 66 66 66 66 2e 0f 1f 84 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369258] BUG: soft lockup - CPU#0 stuck for 21s! [ptlrpcd_1:2001] Mar 9 13:21:40 lustre-utils01 kernel: [116413.369382] Modules linked in: lmv(OF) fld(OF) mgc(OF) lustre(OF) lov(OF) osc(OF) mdc(OF) fid(OF) ksocklnd(OF) ptlrpc(OF) obdclass(OF) lnet(OF) lvfs(OF) libcfs(OF) 8021q(F) garp(F) bridge(F) stp(F) llc(F) acpiphp(F) autofs4(F) binfmt_misc(F) nfsd(F) auth_rpcgss(F) ext2(F) vmwgfx(F) ttm(F) ppdev(F) coretemp(F) nfsv3(F) nfs_acl(F) nfs(F) fscache(F) lockd(F) microcode(F) sunrpc(F) vmw_balloon(F) drm(F) i2c_piix4(F) shpchp(F) psmouse(F) serio_raw(F) mac_hid(F) parport_pc(F) lp(F) parport(F) floppy(F) vmw_pvscsi(F) vmxnet3(F) libcrc32c(F) [last unloaded: ipmi_msghandler] Mar 9 13:21:40 lustre-utils01 kernel: [116413.369416] CPU 0 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369421] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform Mar 9 13:21:40 lustre-utils01 kernel: [116413.369422] RIP: 0010:[] [] ptlrpcd_check+0x11f/0x5a0 [ptlrpc] Mar 9 13:21:40 lustre-utils01 kernel: [116413.369471] RSP: 0000:ffff8804270ddd88 EFLAGS: 00000202 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369472] RAX: ffff880344adb2c0 RBX: ffffffff81e534c0 RCX: ffff8804120508a0 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369474] RDX: ffff8802ec7e2ac0 RSI: 0000000000000000 RDI: ffff880412050a70 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369475] RBP: ffff8804270dddd8 R08: 0000000000000022 R09: 0000000000000000 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369476] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000282 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369477] R13: ffffffff81e534c0 R14: ffffffffa04ff310 R15: ffff8804270ddd78 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369478] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369479] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 9 13:21:40 lustre-utils01 kernel: [116413.369480] CR2: 0000000001b3f006 CR3: 00000002bbc2f000 CR4: 00000000000007f0 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369534] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369556] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369558] Process ptlrpcd_1 (pid: 2001, threadinfo ffff8804270dc000, task ffff880427e9dd00) Mar 9 13:21:40 lustre-utils01 kernel: [116413.369559] Stack: Mar 9 13:21:40 lustre-utils01 kernel: [116413.369560] ffff8804270ddda8 ffff8804270ddda8 0000000000000282 0000000027e9dd00 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369562] ffff8804270dddd8 ffff8804270dde20 ffff880427423980 ffff8804254a8300 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369564] 00000000000000fa ffff880427e9dd00 ffff8804270ddeb8 ffffffffa052ab04 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369579] Call Trace: Mar 9 13:21:40 lustre-utils01 kernel: [116413.369613] [] ptlrpcd+0x1f4/0x3a0 [ptlrpc] Mar 9 13:21:40 lustre-utils01 kernel: [116413.369622] [] ? try_to_wake_up+0x200/0x200 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369644] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 13:21:40 lustre-utils01 kernel: [116413.369647] [] kthread+0xc0/0xd0 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369649] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369653] [] ret_from_fork+0x7c/0xb0 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369655] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 13:21:40 lustre-utils01 kernel: [116413.369656] Code: 1f 44 00 00 8b 43 08 85 c0 0f 85 5d 02 00 00 48 8b 43 30 4c 8d 7b 30 49 39 c7 74 58 4c 8b 20 eb 12 0f 1f 00 4d 39 e7 49 8b 14 24 <74> 47 4c 89 e0 49 89 d4 81 b8 14 fe ff ff 04 de c0 eb 75 e5 48 Mar 9 13:22:09 lustre-utils01 kernel: [116443.220786] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394371224/real 0] req@ffff8801c884f000 x1461982933446032/t0(0) o8->lus13-OST0001-osc-ffff880412525000@172.17.117.144@tcp:28/4 lens 400/544 e 0 to 1 dl 1394371329 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 13:22:09 lustre-utils01 kernel: [116443.220794] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 1117414 previous similar messages Mar 9 13:24:22 lustre-utils01 kernel: [116575.074530] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394371153/real 0] req@ffff880108ca7800 x1461982933443372/t0(0) o400->lus13-OST0000-osc-ffff880412525000@172.17.117.145@tcp:28/4 lens 224/224 e 0 to 1 dl 1394371461 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 13:24:28 lustre-utils01 kernel: [116575.074535] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 115 previous similar messages Mar 9 13:25:40 lustre-utils01 kernel: [116653.248991] Lustre: Evicted from lus13-OST0001_UUID (at 172.17.117.144@tcp) after server handle changed from 0x4927905f62e20a93 to 0x4927905f68e7163e Mar 9 13:25:40 lustre-utils01 kernel: [116653.249001] LustreError: 167-0: lus13-OST0001-osc-ffff880412525000: This client was evicted by lus13-OST0001; in progress operations using this service will fail. Mar 9 13:25:40 lustre-utils01 kernel: [116653.254067] Lustre: Evicted from lus13-OST0002_UUID (at 172.17.117.144@tcp) after server handle changed from 0x4927905f62e20aa8 to 0x4927905f68e71653 Mar 9 13:25:42 lustre-utils01 kernel: [116655.199436] Lustre: lus13-OST0001-osc-ffff880412525000: Connection restored to lus13-OST0001 (at 172.17.117.144@tcp) Mar 9 13:29:30 lustre-utils01 kernel: [116883.638227] LustreError: 11-0: lus13-OST0008-osc-ffff880412525000: Communicating with 172.17.117.145@tcp, operation obd_ping failed with -107. Mar 9 13:29:30 lustre-utils01 kernel: [116883.638464] LustreError: Skipped 23 previous similar messages Mar 9 13:29:30 lustre-utils01 kernel: [116883.638556] Lustre: lus13-OST0008-osc-ffff880412525000: Connection to lus13-OST0008 (at 172.17.117.145@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 13:29:30 lustre-utils01 kernel: [116883.638558] Lustre: Skipped 27 previous similar messages Mar 9 13:29:31 lustre-utils01 kernel: [116884.647183] LustreError: 167-0: lus13-OST0009-osc-ffff880412525000: This client was evicted by lus13-OST0009; in progress operations using this service will fail. Mar 9 13:29:31 lustre-utils01 kernel: [116884.647403] LustreError: Skipped 17 previous similar messages Mar 9 13:29:31 lustre-utils01 kernel: [116884.857991] LustreError: 11-0: lus13-OST0004-osc-ffff880412525000: Communicating with 172.17.117.144@tcp, operation obd_ping failed with -107. Mar 9 13:29:31 lustre-utils01 kernel: [116884.858204] LustreError: Skipped 11 previous similar messages Mar 9 13:29:31 lustre-utils01 kernel: [116884.858480] Lustre: lus13-OST0007-osc-ffff880412525000: Connection restored to lus13-OST0007 (at 172.17.117.145@tcp) Mar 9 13:29:31 lustre-utils01 kernel: [116884.858482] Lustre: Skipped 35 previous similar messages Mar 9 13:30:16 lustre-utils01 kernel: [116929.014286] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394371585/real 0] req@ffff88018df8c800 x1461982933454492/t0(0) o400->lus13-OST000c-osc-ffff880412525000@172.17.117.146@tcp:28/4 lens 224/224 e 0 to 1 dl 1394371815 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 13:30:16 lustre-utils01 kernel: [116929.014293] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 77 previous similar messages Mar 9 13:30:41 lustre-utils01 kernel: [116954.490115] LustreError: 167-0: lus13-OST0024-osc-ffff880412525000: This client was evicted by lus13-OST0024; in progress operations using this service will fail. Mar 9 13:30:41 lustre-utils01 kernel: [116954.490383] LustreError: Skipped 7 previous similar messages Mar 9 13:30:45 lustre-utils01 kernel: [116958.680330] LustreError: 11-0: lus13-OST0017-osc-ffff880412525000: Communicating with 172.17.117.148@tcp, operation ldlm_enqueue failed with -107. Mar 9 13:30:45 lustre-utils01 kernel: [116958.680524] LustreError: Skipped 5 previous similar messages Mar 9 13:30:45 lustre-utils01 kernel: [116958.680632] Lustre: lus13-OST0017-osc-ffff880412525000: Connection to lus13-OST0017 (at 172.17.117.148@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 13:30:45 lustre-utils01 kernel: [116958.680635] Lustre: Skipped 17 previous similar messages Mar 9 13:31:15 lustre-utils01 kernel: [116988.538339] LustreError: 11-0: lus13-OST001c-osc-ffff880412525000: Communicating with 172.17.117.149@tcp, operation ldlm_enqueue failed with -107. Mar 9 13:31:38 lustre-utils01 kernel: [117011.756113] LustreError: 11-0: lus13-OST0017-osc-ffff880412525000: Communicating with 172.17.117.148@tcp, operation ldlm_enqueue failed with -107. Mar 9 13:31:38 lustre-utils01 kernel: [117011.756298] LustreError: Skipped 1 previous similar message Mar 9 13:31:47 lustre-utils01 kernel: [117020.558231] LustreError: 167-0: lus13-OST000c-osc-ffff880412525000: This client was evicted by lus13-OST000c; in progress operations using this service will fail. Mar 9 13:31:47 lustre-utils01 kernel: [117020.558436] LustreError: Skipped 3 previous similar messages Mar 9 13:32:06 lustre-utils01 kernel: [117039.922764] Lustre: Evicted from lus13-OST0012_UUID (at 172.17.117.147@tcp) after server handle changed from 0x817cdc2fd5c95999 to 0x817cdc2fd6a1ce9c Mar 9 13:32:07 lustre-utils01 kernel: [117039.922799] Lustre: Skipped 16 previous similar messages Mar 9 13:32:07 lustre-utils01 kernel: [117039.922805] LustreError: 167-0: lus13-OST0012-osc-ffff880412525000: This client was evicted by lus13-OST0012; in progress operations using this service will fail. Mar 9 13:32:07 lustre-utils01 kernel: [117039.923019] LustreError: Skipped 3 previous similar messages Mar 9 13:32:09 lustre-utils01 kernel: [117042.051697] LustreError: 11-0: lus13-OST001c-osc-ffff880412525000: Communicating with 172.17.117.149@tcp, operation ldlm_enqueue failed with -107. Mar 9 13:32:09 lustre-utils01 kernel: [117042.051881] LustreError: Skipped 2 previous similar messages Mar 9 13:32:40 lustre-utils01 kernel: [117073.278489] LustreError: 11-0: lus13-OST0018-osc-ffff880412525000: Communicating with 172.17.117.148@tcp, operation ldlm_enqueue failed with -107. Mar 9 13:32:40 lustre-utils01 kernel: [117073.278684] LustreError: Skipped 3 previous similar messages Mar 9 13:32:41 lustre-utils01 kernel: [117074.328961] LustreError: 167-0: lus13-OST0017-osc-ffff880412525000: This client was evicted by lus13-OST0017; in progress operations using this service will fail. Mar 9 13:32:41 lustre-utils01 kernel: [117074.329221] LustreError: Skipped 1 previous similar message Mar 9 13:32:57 lustre-utils01 kernel: [117090.070548] Lustre: Evicted from lus13-OST0015_UUID (at 172.17.117.148@tcp) after server handle changed from 0x4fe201e0e1f79cc4 to 0x4fe201e0e8044f2d Mar 9 13:32:57 lustre-utils01 kernel: [117090.070553] Lustre: Skipped 1 previous similar message Mar 9 13:33:22 lustre-utils01 kernel: [117114.958984] Lustre: Evicted from lus13-OST0021_UUID (at 172.17.117.150@tcp) after server handle changed from 0x41126d0eb06f24dd to 0x41126d0eb06f3139 Mar 9 13:33:22 lustre-utils01 kernel: [117114.958991] Lustre: Skipped 1 previous similar message Mar 9 17:43:27 lustre-utils01 kernel: [132112.739765] warn_alloc_failed: 72 callbacks suppressed Mar 9 17:43:27 lustre-utils01 kernel: [132112.739776] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:43:27 lustre-utils01 kernel: [132112.739786] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:43:27 lustre-utils01 kernel: [132112.739787] Call Trace: Mar 9 17:43:27 lustre-utils01 kernel: [132112.739823] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:43:27 lustre-utils01 kernel: [132112.739826] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:43:27 lustre-utils01 kernel: [132112.739834] [] alloc_pages_current+0xba/0x170 Mar 9 17:43:27 lustre-utils01 kernel: [132112.739840] [] allocate_slab+0x190/0x1a0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.739842] [] new_slab+0x30/0x1a0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.739855] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:43:27 lustre-utils01 kernel: [132112.739884] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 17:43:27 lustre-utils01 kernel: [132112.739891] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 17:43:27 lustre-utils01 kernel: [132112.739893] [] kmem_cache_alloc_trace+0x132/0x140 Mar 9 17:43:27 lustre-utils01 kernel: [132112.739899] [] ? lnet_md_alloc.isra.2+0x74/0x280 [lnet] Mar 9 17:43:27 lustre-utils01 kernel: [132112.739905] [] LNetPut+0x122/0x900 [lnet] Mar 9 17:43:27 lustre-utils01 kernel: [132112.739930] [] ? cfs_percpt_unlock+0x42/0xd0 [libcfs] Mar 9 17:43:27 lustre-utils01 kernel: [132112.739989] [] ptl_send_buf+0x12f/0x5d0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.739996] [] ? LNetMDAttach+0x2db/0x370 [lnet] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740006] [] ? read_tsc+0x9/0x20 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740027] [] ptl_send_rpc+0x522/0xc20 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740047] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740071] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740095] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740126] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740150] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740171] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740194] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740219] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.740231] [] kthread+0xc0/0xd0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740234] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740246] [] ret_from_fork+0x7c/0xb0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740249] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740250] Mem-Info: Mar 9 17:43:27 lustre-utils01 kernel: [132112.740251] Node 0 DMA per-cpu: Mar 9 17:43:27 lustre-utils01 kernel: [132112.740254] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740254] Node 0 DMA32 per-cpu: Mar 9 17:43:27 lustre-utils01 kernel: [132112.740256] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740257] Node 0 Normal per-cpu: Mar 9 17:43:27 lustre-utils01 kernel: [132112.740258] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740261] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740261] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740261] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740261] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740261] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740261] free_cma:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740264] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:43:27 lustre-utils01 kernel: [132112.740268] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740271] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:43:27 lustre-utils01 kernel: [132112.740274] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740276] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:43:27 lustre-utils01 kernel: [132112.740280] lowmem_reserve[]: 0 0 0 0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740282] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.740289] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.740297] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 13*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41032kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.740303] 106183 total pagecache pages Mar 9 17:43:27 lustre-utils01 kernel: [132112.740305] 2434 pages in swap cache Mar 9 17:43:27 lustre-utils01 kernel: [132112.740306] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:43:27 lustre-utils01 kernel: [132112.740307] Free swap = 38730268kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.740308] Total swap = 38793204kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.790668] 4194288 pages RAM Mar 9 17:43:27 lustre-utils01 kernel: [132112.790673] 85481 pages reserved Mar 9 17:43:27 lustre-utils01 kernel: [132112.790674] 359924 pages shared Mar 9 17:43:27 lustre-utils01 kernel: [132112.790674] 4016447 pages non-shared Mar 9 17:43:27 lustre-utils01 kernel: [132112.790679] __slab_alloc: 72 callbacks suppressed Mar 9 17:43:27 lustre-utils01 kernel: [132112.790680] SLUB: Unable to allocate memory on node -1 (gfp=0x40) Mar 9 17:43:27 lustre-utils01 kernel: [132112.790682] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.790684] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.790840] LNetError: 2000:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: out of memory at /var/lib/dkms/lustre-client-modules-2.5.0/2.5.0/build/lnet/include/lnet/lib-lnet.h:506 (tried to alloc '(msg)' = 352) Mar 9 17:43:27 lustre-utils01 kernel: [132112.791355] LNetError: 2000:0:(lib-lnet.h:506:lnet_msg_alloc()) Skipped 38 previous similar messages Mar 9 17:43:27 lustre-utils01 kernel: [132112.791484] LNetError: 2000:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: 715972943 total bytes allocated by lnet Mar 9 17:43:27 lustre-utils01 kernel: [132112.791618] LNetError: 2000:0:(lib-lnet.h:506:lnet_msg_alloc()) Skipped 38 previous similar messages Mar 9 17:43:27 lustre-utils01 kernel: [132112.791753] LNetError: 2000:0:(lib-move.c:2138:LNetPut()) Dropping PUT to 12345-172.17.117.144@tcp: ENOMEM on lnet_msg_t Mar 9 17:43:27 lustre-utils01 kernel: [132112.791895] LNetError: 2000:0:(lib-move.c:2138:LNetPut()) Skipped 38 previous similar messages Mar 9 17:43:27 lustre-utils01 kernel: [132112.792022] LustreError: 2000:0:(niobuf.c:96:ptl_send_buf()) LNetPut(12345-172.17.117.144@tcp, 17, 1461983178195512) failed: -12 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792178] LustreError: 2000:0:(niobuf.c:96:ptl_send_buf()) Skipped 38 previous similar messages Mar 9 17:43:27 lustre-utils01 kernel: [132112.792334] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1394387006/real 1394387006] req@ffff88002da1ec00 x1461983178195512/t0(0) o103->lus13-OST0003-osc-ffff880412525000@172.17.117.144@tcp:17/18 lens 344/224 e 0 to 1 dl 1394387236 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792336] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 26 previous similar messages Mar 9 17:43:27 lustre-utils01 kernel: [132112.792347] Lustre: lus13-OST0003-osc-ffff880412525000: Connection to lus13-OST0003 (at 172.17.117.144@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 17:43:27 lustre-utils01 kernel: [132112.792348] Lustre: Skipped 3 previous similar messages Mar 9 17:43:27 lustre-utils01 kernel: [132112.792373] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792375] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:43:27 lustre-utils01 kernel: [132112.792377] Call Trace: Mar 9 17:43:27 lustre-utils01 kernel: [132112.792388] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792390] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792404] [] ? console_unlock.part.8+0x22e/0x280 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792410] [] alloc_pages_current+0xba/0x170 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792417] [] allocate_slab+0x190/0x1a0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792419] [] new_slab+0x30/0x1a0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792423] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792469] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792472] [] ? check_preempt_curr+0x84/0xa0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792474] [] __kmalloc+0x162/0x190 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792476] [] ? ttwu_do_activate.constprop.86+0x66/0x70 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792498] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792519] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792539] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792560] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792563] [] ? default_wake_function+0x12/0x20 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792583] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792604] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792625] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792645] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792664] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792686] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792707] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792709] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792730] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:43:27 lustre-utils01 kernel: [132112.792734] [] kthread+0xc0/0xd0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792736] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792739] [] ret_from_fork+0x7c/0xb0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792741] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792742] Mem-Info: Mar 9 17:43:27 lustre-utils01 kernel: [132112.792744] Node 0 DMA per-cpu: Mar 9 17:43:27 lustre-utils01 kernel: [132112.792746] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792747] Node 0 DMA32 per-cpu: Mar 9 17:43:27 lustre-utils01 kernel: [132112.792757] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792758] Node 0 Normal per-cpu: Mar 9 17:43:27 lustre-utils01 kernel: [132112.792760] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792762] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792762] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792762] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792762] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792762] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792762] free_cma:0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792765] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:43:27 lustre-utils01 kernel: [132112.792769] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792771] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:43:27 lustre-utils01 kernel: [132112.792775] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792777] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:43:27 lustre-utils01 kernel: [132112.792781] lowmem_reserve[]: 0 0 0 0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792782] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.792790] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.792797] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 13*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41032kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.792804] 106183 total pagecache pages Mar 9 17:43:27 lustre-utils01 kernel: [132112.792805] 2434 pages in swap cache Mar 9 17:43:27 lustre-utils01 kernel: [132112.792807] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:43:27 lustre-utils01 kernel: [132112.792808] Free swap = 38730268kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.792808] Total swap = 38793204kB Mar 9 17:43:27 lustre-utils01 kernel: [132112.841877] 4194288 pages RAM Mar 9 17:43:27 lustre-utils01 kernel: [132112.841881] 85481 pages reserved Mar 9 17:43:27 lustre-utils01 kernel: [132112.841882] 359916 pages shared Mar 9 17:43:27 lustre-utils01 kernel: [132112.841883] 4016447 pages non-shared Mar 9 17:43:27 lustre-utils01 kernel: [132112.841887] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 17:43:27 lustre-utils01 kernel: [132112.841889] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.841891] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:43:27 lustre-utils01 kernel: [132112.842015] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:43:27 lustre-utils01 kernel: [132112.842018] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:43:27 lustre-utils01 kernel: [132112.842019] Call Trace: Mar 9 17:43:27 lustre-utils01 kernel: [132112.842030] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:43:27 lustre-utils01 kernel: [132112.842056] [] ? number.isra.2+0x355/0x390 Mar 9 17:43:27 lustre-utils01 kernel: [132112.842058] [] ? put_dec+0x72/0x90 Mar 9 17:43:27 lustre-utils01 kernel: [132112.842060] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:43:27 lustre-utils01 kernel: [132112.842062] [] ? put_dec+0x72/0x90 Mar 9 17:43:27 lustre-utils01 kernel: [132112.842084] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 17:43:27 lustre-utils01 kernel: [132112.842088] [] alloc_pages_current+0xba/0x170 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842091] [] allocate_slab+0x190/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842093] [] new_slab+0x30/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842096] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842158] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842168] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842178] [] __kmalloc+0x162/0x190 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842201] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842223] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842243] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842264] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842268] [] ? default_wake_function+0x12/0x20 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842288] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842309] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842330] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842350] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842369] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842391] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842412] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842415] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842436] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.842439] [] kthread+0xc0/0xd0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842441] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842445] [] ret_from_fork+0x7c/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842447] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842448] Mem-Info: Mar 9 17:44:04 lustre-utils01 kernel: [132112.842450] Node 0 DMA per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.842452] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842453] Node 0 DMA32 per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.842454] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842455] Node 0 Normal per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.842456] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842459] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842459] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842459] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842459] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842459] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842459] free_cma:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842462] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:44:04 lustre-utils01 kernel: [132112.842467] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842469] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132112.842473] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842475] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132112.842479] lowmem_reserve[]: 0 0 0 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842481] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.842489] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.842497] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 13*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41032kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.842511] 106183 total pagecache pages Mar 9 17:44:04 lustre-utils01 kernel: [132112.842513] 2434 pages in swap cache Mar 9 17:44:04 lustre-utils01 kernel: [132112.842514] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:44:04 lustre-utils01 kernel: [132112.842515] Free swap = 38730268kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.842516] Total swap = 38793204kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.891561] 4194288 pages RAM Mar 9 17:44:04 lustre-utils01 kernel: [132112.891565] 85481 pages reserved Mar 9 17:44:04 lustre-utils01 kernel: [132112.891566] 359916 pages shared Mar 9 17:44:04 lustre-utils01 kernel: [132112.891567] 4016447 pages non-shared Mar 9 17:44:04 lustre-utils01 kernel: [132112.891572] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 17:44:04 lustre-utils01 kernel: [132112.891574] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891576] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891759] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891762] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:44:04 lustre-utils01 kernel: [132112.891764] Call Trace: Mar 9 17:44:04 lustre-utils01 kernel: [132112.891774] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891779] [] ? number.isra.2+0x355/0x390 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891781] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891783] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891785] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891808] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132112.891812] [] alloc_pages_current+0xba/0x170 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891815] [] allocate_slab+0x190/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891817] [] new_slab+0x30/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891820] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891864] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.891870] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132112.891873] [] __kmalloc+0x162/0x190 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891895] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.891916] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.891937] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.891958] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.891962] [] ? default_wake_function+0x12/0x20 Mar 9 17:44:04 lustre-utils01 kernel: [132112.891981] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.892003] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.892024] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.892044] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.892063] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.892084] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.892106] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.892108] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892131] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.892136] [] kthread+0xc0/0xd0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892139] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892153] [] ret_from_fork+0x7c/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892157] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892158] Mem-Info: Mar 9 17:44:04 lustre-utils01 kernel: [132112.892160] Node 0 DMA per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.892162] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892163] Node 0 DMA32 per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.892164] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892165] Node 0 Normal per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.892166] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892169] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892169] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892169] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892169] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892169] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892169] free_cma:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892172] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:44:04 lustre-utils01 kernel: [132112.892177] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892179] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132112.892183] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892185] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132112.892189] lowmem_reserve[]: 0 0 0 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892190] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.892198] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.892205] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 13*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41032kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.892212] 106183 total pagecache pages Mar 9 17:44:04 lustre-utils01 kernel: [132112.892213] 2434 pages in swap cache Mar 9 17:44:04 lustre-utils01 kernel: [132112.892215] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:44:04 lustre-utils01 kernel: [132112.892216] Free swap = 38730268kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.892216] Total swap = 38793204kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.941110] 4194288 pages RAM Mar 9 17:44:04 lustre-utils01 kernel: [132112.941115] 85481 pages reserved Mar 9 17:44:04 lustre-utils01 kernel: [132112.941116] 359900 pages shared Mar 9 17:44:04 lustre-utils01 kernel: [132112.941116] 4016447 pages non-shared Mar 9 17:44:04 lustre-utils01 kernel: [132112.941122] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 17:44:04 lustre-utils01 kernel: [132112.941125] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941127] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941296] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941299] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:44:04 lustre-utils01 kernel: [132112.941300] Call Trace: Mar 9 17:44:04 lustre-utils01 kernel: [132112.941312] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941317] [] ? number.isra.2+0x355/0x390 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941319] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941321] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941323] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941346] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941349] [] alloc_pages_current+0xba/0x170 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941352] [] allocate_slab+0x190/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941354] [] new_slab+0x30/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941357] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941405] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941413] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941415] [] __kmalloc+0x162/0x190 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941437] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941459] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941478] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941500] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941504] [] ? _raw_spin_unlock_irqrestore+0x19/0x30 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941524] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941545] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941582] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941603] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941623] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941644] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941665] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941669] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941691] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.941694] [] kthread+0xc0/0xd0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941696] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941699] [] ret_from_fork+0x7c/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941702] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941703] Mem-Info: Mar 9 17:44:04 lustre-utils01 kernel: [132112.941704] Node 0 DMA per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.941706] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941707] Node 0 DMA32 per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.941709] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941709] Node 0 Normal per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.941711] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941714] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941714] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941714] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941714] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941714] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941714] free_cma:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941717] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:44:04 lustre-utils01 kernel: [132112.941721] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941723] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132112.941727] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941729] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132112.941733] lowmem_reserve[]: 0 0 0 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941735] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.941743] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.941750] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 13*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41032kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.941757] 106183 total pagecache pages Mar 9 17:44:04 lustre-utils01 kernel: [132112.941758] 2434 pages in swap cache Mar 9 17:44:04 lustre-utils01 kernel: [132112.941760] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:44:04 lustre-utils01 kernel: [132112.941761] Free swap = 38730268kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.941761] Total swap = 38793204kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.990587] 4194288 pages RAM Mar 9 17:44:04 lustre-utils01 kernel: [132112.990591] 85481 pages reserved Mar 9 17:44:04 lustre-utils01 kernel: [132112.990592] 359900 pages shared Mar 9 17:44:04 lustre-utils01 kernel: [132112.990593] 4016447 pages non-shared Mar 9 17:44:04 lustre-utils01 kernel: [132112.990598] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 17:44:04 lustre-utils01 kernel: [132112.990600] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.990602] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.990980] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:44:04 lustre-utils01 kernel: [132112.990983] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:44:04 lustre-utils01 kernel: [132112.990984] Call Trace: Mar 9 17:44:04 lustre-utils01 kernel: [132112.990995] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:44:04 lustre-utils01 kernel: [132112.990998] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991003] [] ? number.isra.2+0x355/0x390 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991011] [] alloc_pages_current+0xba/0x170 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991015] [] allocate_slab+0x190/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991016] [] new_slab+0x30/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991021] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991036] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991042] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991045] [] kmem_cache_alloc_trace+0x132/0x140 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991051] [] ? lnet_md_alloc.isra.2+0x74/0x280 [lnet] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991057] [] LNetPut+0x122/0x900 [lnet] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991078] [] ? cfs_percpt_unlock+0x42/0xd0 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991126] [] ptl_send_buf+0x12f/0x5d0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991134] [] ? LNetMDAttach+0x2db/0x370 [lnet] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991137] [] ? read_tsc+0x9/0x20 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991157] [] ptl_send_rpc+0x522/0xc20 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991185] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991209] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991232] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991252] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991271] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991292] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991314] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991317] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991339] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132112.991342] [] kthread+0xc0/0xd0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991344] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991348] [] ret_from_fork+0x7c/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991350] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991351] Mem-Info: Mar 9 17:44:04 lustre-utils01 kernel: [132112.991353] Node 0 DMA per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.991355] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991356] Node 0 DMA32 per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.991357] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991358] Node 0 Normal per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132112.991359] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991362] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991362] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991362] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991362] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991362] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991362] free_cma:0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991365] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:44:04 lustre-utils01 kernel: [132112.991370] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991372] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132112.991376] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991378] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132112.991401] lowmem_reserve[]: 0 0 0 0 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991403] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.991411] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.991418] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 12*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41000kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.991425] 106183 total pagecache pages Mar 9 17:44:04 lustre-utils01 kernel: [132112.991427] 2434 pages in swap cache Mar 9 17:44:04 lustre-utils01 kernel: [132112.991428] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:44:04 lustre-utils01 kernel: [132112.991429] Free swap = 38730268kB Mar 9 17:44:04 lustre-utils01 kernel: [132112.991429] Total swap = 38793204kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.040589] 4194288 pages RAM Mar 9 17:44:04 lustre-utils01 kernel: [132113.040592] 85481 pages reserved Mar 9 17:44:04 lustre-utils01 kernel: [132113.040593] 360020 pages shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.040594] 4016447 pages non-shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.040599] SLUB: Unable to allocate memory on node -1 (gfp=0x40) Mar 9 17:44:04 lustre-utils01 kernel: [132113.040601] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.040603] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.040754] LNetError: 2000:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: out of memory at /var/lib/dkms/lustre-client-modules-2.5.0/2.5.0/build/lnet/include/lnet/lib-lnet.h:506 (tried to alloc '(msg)' = 352) Mar 9 17:44:04 lustre-utils01 kernel: [132113.040987] LNetError: 2000:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: 715974351 total bytes allocated by lnet Mar 9 17:44:04 lustre-utils01 kernel: [132113.041134] LNetError: 2000:0:(lib-move.c:2138:LNetPut()) Dropping PUT to 12345-172.17.117.149@tcp: ENOMEM on lnet_msg_t Mar 9 17:44:04 lustre-utils01 kernel: [132113.041280] LustreError: 2000:0:(niobuf.c:96:ptl_send_buf()) LNetPut(12345-172.17.117.149@tcp, 17, 1461983178195568) failed: -12 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041481] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041484] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:44:04 lustre-utils01 kernel: [132113.041485] Call Trace: Mar 9 17:44:04 lustre-utils01 kernel: [132113.041497] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041501] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041504] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041506] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041528] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041532] [] alloc_pages_current+0xba/0x170 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041535] [] allocate_slab+0x190/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041536] [] new_slab+0x30/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041541] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041586] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041593] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041595] [] __kmalloc+0x162/0x190 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041617] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041639] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041659] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041680] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041683] [] ? default_wake_function+0x12/0x20 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041703] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041724] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041746] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041765] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041784] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041806] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041827] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041847] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041869] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.041873] [] kthread+0xc0/0xd0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041875] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041880] [] ret_from_fork+0x7c/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041882] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041883] Mem-Info: Mar 9 17:44:04 lustre-utils01 kernel: [132113.041885] Node 0 DMA per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.041887] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041888] Node 0 DMA32 per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.041889] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041890] Node 0 Normal per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.041891] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041894] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041894] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041894] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041894] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041894] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041894] free_cma:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041897] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:44:04 lustre-utils01 kernel: [132113.041901] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041904] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132113.041907] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041909] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132113.041913] lowmem_reserve[]: 0 0 0 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041915] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.041922] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.041930] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 12*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41000kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.041936] 106183 total pagecache pages Mar 9 17:44:04 lustre-utils01 kernel: [132113.041938] 2434 pages in swap cache Mar 9 17:44:04 lustre-utils01 kernel: [132113.041939] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:44:04 lustre-utils01 kernel: [132113.041940] Free swap = 38730268kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.041941] Total swap = 38793204kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.091246] 4194288 pages RAM Mar 9 17:44:04 lustre-utils01 kernel: [132113.091250] 85481 pages reserved Mar 9 17:44:04 lustre-utils01 kernel: [132113.091251] 360012 pages shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.091252] 4016447 pages non-shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.091257] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 17:44:04 lustre-utils01 kernel: [132113.091259] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091261] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091459] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091462] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:44:04 lustre-utils01 kernel: [132113.091464] Call Trace: Mar 9 17:44:04 lustre-utils01 kernel: [132113.091475] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091480] [] ? number.isra.2+0x355/0x390 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091481] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091484] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091485] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091509] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091512] [] alloc_pages_current+0xba/0x170 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091515] [] allocate_slab+0x190/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091517] [] new_slab+0x30/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091521] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091565] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091572] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091574] [] __kmalloc+0x162/0x190 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091596] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091617] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091637] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091659] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091662] [] ? default_wake_function+0x12/0x20 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091682] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091704] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091725] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091744] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091764] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091785] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091806] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091809] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091830] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.091833] [] kthread+0xc0/0xd0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091835] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091839] [] ret_from_fork+0x7c/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091841] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091842] Mem-Info: Mar 9 17:44:04 lustre-utils01 kernel: [132113.091844] Node 0 DMA per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.091846] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091847] Node 0 DMA32 per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.091848] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091849] Node 0 Normal per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.091850] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091853] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091853] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091853] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091853] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091853] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091853] free_cma:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091856] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:44:04 lustre-utils01 kernel: [132113.091860] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091863] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132113.091866] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091868] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132113.091872] lowmem_reserve[]: 0 0 0 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091874] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.091882] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.091889] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 12*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41000kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.091896] 106183 total pagecache pages Mar 9 17:44:04 lustre-utils01 kernel: [132113.091897] 2434 pages in swap cache Mar 9 17:44:04 lustre-utils01 kernel: [132113.091898] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:44:04 lustre-utils01 kernel: [132113.091899] Free swap = 38730268kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.091900] Total swap = 38793204kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.141085] 4194288 pages RAM Mar 9 17:44:04 lustre-utils01 kernel: [132113.141090] 85481 pages reserved Mar 9 17:44:04 lustre-utils01 kernel: [132113.141091] 360004 pages shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.141091] 4016447 pages non-shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.141097] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 17:44:04 lustre-utils01 kernel: [132113.141099] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141101] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141198] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141200] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:44:04 lustre-utils01 kernel: [132113.141202] Call Trace: Mar 9 17:44:04 lustre-utils01 kernel: [132113.141213] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141218] [] ? number.isra.2+0x355/0x390 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141219] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141222] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141224] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141247] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141251] [] alloc_pages_current+0xba/0x170 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141254] [] allocate_slab+0x190/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141256] [] new_slab+0x30/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141259] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141305] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141312] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141314] [] __kmalloc+0x162/0x190 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141336] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141358] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141378] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141399] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141403] [] ? default_wake_function+0x12/0x20 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141423] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141444] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141465] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141485] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141504] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141526] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141547] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141549] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141571] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.141574] [] kthread+0xc0/0xd0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141576] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141580] [] ret_from_fork+0x7c/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141582] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141583] Mem-Info: Mar 9 17:44:04 lustre-utils01 kernel: [132113.141584] Node 0 DMA per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.141587] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141588] Node 0 DMA32 per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.141589] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141590] Node 0 Normal per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.141591] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141594] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141594] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141594] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141594] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141594] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141594] free_cma:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141597] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:44:04 lustre-utils01 kernel: [132113.141601] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141604] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132113.141608] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141609] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132113.141613] lowmem_reserve[]: 0 0 0 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141615] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.141623] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.141630] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 12*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41000kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.141636] 106183 total pagecache pages Mar 9 17:44:04 lustre-utils01 kernel: [132113.141638] 2434 pages in swap cache Mar 9 17:44:04 lustre-utils01 kernel: [132113.141639] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:44:04 lustre-utils01 kernel: [132113.141640] Free swap = 38730268kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.141641] Total swap = 38793204kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.190772] 4194288 pages RAM Mar 9 17:44:04 lustre-utils01 kernel: [132113.190777] 85481 pages reserved Mar 9 17:44:04 lustre-utils01 kernel: [132113.190777] 360004 pages shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.190778] 4016447 pages non-shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.190783] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 17:44:04 lustre-utils01 kernel: [132113.190785] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.190787] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.190978] ptlrpcd_0: page allocation failure: order:0, mode:0x200040 Mar 9 17:44:04 lustre-utils01 kernel: [132113.190982] Pid: 2000, comm: ptlrpcd_0 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 17:44:04 lustre-utils01 kernel: [132113.190984] Call Trace: Mar 9 17:44:04 lustre-utils01 kernel: [132113.190997] [] warn_alloc_failed+0xf6/0x150 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191003] [] ? number.isra.2+0x355/0x390 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191006] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191010] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191011] [] ? put_dec+0x72/0x90 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191035] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191039] [] alloc_pages_current+0xba/0x170 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191042] [] allocate_slab+0x190/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191043] [] new_slab+0x30/0x1a0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191047] [] __slab_alloc+0x17d/0x2b2 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191103] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191110] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191113] [] __kmalloc+0x162/0x190 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191135] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191156] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191176] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191197] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191201] [] ? default_wake_function+0x12/0x20 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191221] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191242] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191263] [] ? null_free_reqbuf+0x124/0x2e0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191283] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191302] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191324] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191345] [] ptlrpcd+0x18f/0x3a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191348] [] ? try_to_wake_up+0x200/0x200 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191369] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 17:44:04 lustre-utils01 kernel: [132113.191372] [] kthread+0xc0/0xd0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191374] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191378] [] ret_from_fork+0x7c/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191380] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191381] Mem-Info: Mar 9 17:44:04 lustre-utils01 kernel: [132113.191383] Node 0 DMA per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.191385] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191386] Node 0 DMA32 per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.191387] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191388] Node 0 Normal per-cpu: Mar 9 17:44:04 lustre-utils01 kernel: [132113.191390] CPU 0: hi: 186, btch: 31 usd: 40 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191393] active_anon:19817 inactive_anon:30165 isolated_anon:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191393] active_file:70030 inactive_file:32838 isolated_file:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191393] unevictable:15607 dirty:0 writeback:0 unstable:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191393] free:29680 slab_reclaimable:108469 slab_unreclaimable:3597142 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191393] mapped:29026 shmem:33 pagetables:1446 bounce:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191393] free_cma:0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191396] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 17:44:04 lustre-utils01 kernel: [132113.191400] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191402] Node 0 DMA32 free:61784kB min:12604kB low:15752kB high:18904kB active_anon:2620kB inactive_anon:2812kB active_file:6868kB inactive_file:13948kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:0kB writeback:0kB mapped:10288kB shmem:0kB slab_reclaimable:58368kB slab_unreclaimable:2903388kB kernel_stack:8kB pagetables:52kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132113.191406] lowmem_reserve[]: 0 0 13104 13104 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191408] Node 0 Normal free:41032kB min:54912kB low:68640kB high:82368kB active_anon:76648kB inactive_anon:117848kB active_file:273252kB inactive_file:117404kB unevictable:62428kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:62428kB dirty:0kB writeback:0kB mapped:105816kB shmem:132kB slab_reclaimable:375508kB slab_unreclaimable:11485176kB kernel_stack:2240kB pagetables:5732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 17:44:04 lustre-utils01 kernel: [132113.191411] lowmem_reserve[]: 0 0 0 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191413] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.191421] Node 0 DMA32: 2*4kB (UM) 2*8kB (EM) 352*16kB (UE) 106*32kB (UE) 494*64kB (UEM) 131*128kB (UEM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61784kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.191428] Node 0 Normal: 2132*4kB (UEM) 109*8kB (UEM) 1695*16kB (UM) 12*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 41000kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.191435] 106183 total pagecache pages Mar 9 17:44:04 lustre-utils01 kernel: [132113.191436] 2434 pages in swap cache Mar 9 17:44:04 lustre-utils01 kernel: [132113.191456] Swap cache stats: add 18906, delete 16472, find 23839/24218 Mar 9 17:44:04 lustre-utils01 kernel: [132113.191457] Free swap = 38730268kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.191458] Total swap = 38793204kB Mar 9 17:44:04 lustre-utils01 kernel: [132113.240510] 4194288 pages RAM Mar 9 17:44:04 lustre-utils01 kernel: [132113.240514] 85481 pages reserved Mar 9 17:44:04 lustre-utils01 kernel: [132113.240514] 359988 pages shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.240515] 4016447 pages non-shared Mar 9 17:44:04 lustre-utils01 kernel: [132113.240521] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 17:44:04 lustre-utils01 kernel: [132113.240523] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.240525] node 0: slabs: 22665, objs: 181320, free: 0 Mar 9 17:44:04 lustre-utils01 kernel: [132113.291235] LNetError: 1996:0:(lib-move.c:1903:lnet_parse()) 172.17.117.151@tcp, src 172.17.117.151@tcp: Dropping PUT (out of memory) Mar 9 17:44:04 lustre-utils01 kernel: [132113.855099] Lustre: lus13-MDT0000-mdc-ffff880412525000: Connection restored to lus13-MDT0000 (at 172.17.117.153@tcp) Mar 9 17:44:04 lustre-utils01 kernel: [132113.855104] Lustre: Skipped 25 previous similar messages Mar 9 17:44:04 lustre-utils01 kernel: [132137.544715] LustreError: 11-0: lus13-OST0005-osc-ffff880412525000: Communicating with 172.17.117.145@tcp, operation ost_connect failed with -16. Mar 9 17:44:04 lustre-utils01 kernel: [132137.544907] LustreError: Skipped 6 previous similar messages Mar 9 17:44:16 lustre-utils01 kernel: [132162.271048] Lustre: lus13-OST0005-osc-ffff880412525000: Connection restored to lus13-OST0005 (at 172.17.117.145@tcp) Mar 9 17:44:16 lustre-utils01 kernel: [132162.271053] Lustre: Skipped 6 previous similar messages Mar 9 17:53:27 lustre-utils01 kernel: [132713.525602] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394387005/real 1394387006] req@ffff8802b2ca8400 x1461983178156400/t0(0) o103->lus13-OST0024-osc-ffff880412525000@172.17.117.151@tcp:17/18 lens 344/224 e 0 to 1 dl 1394387235 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 17:53:27 lustre-utils01 kernel: [132713.525608] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 33 previous similar messages Mar 9 17:53:27 lustre-utils01 kernel: [132713.525630] Lustre: lus13-OST0024-osc-ffff880412525000: Connection to lus13-OST0024 (at 172.17.117.151@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 17:53:27 lustre-utils01 kernel: [132713.525631] Lustre: Skipped 21 previous similar messages Mar 9 17:53:45 lustre-utils01 kernel: [132731.241952] Lustre: lus13-OST0024-osc-ffff880412525000: Connection restored to lus13-OST0024 (at 172.17.117.151@tcp) Mar 9 17:53:45 lustre-utils01 kernel: [132731.241957] Lustre: Skipped 14 previous similar messages Mar 9 22:04:44 lustre-utils01 kernel: [147697.869905] warn_alloc_failed: 27 callbacks suppressed Mar 9 22:04:44 lustre-utils01 kernel: [147697.869916] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147697.869922] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147697.869924] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147697.869956] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147697.869959] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147697.869966] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147697.869971] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.869973] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.869986] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870011] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870018] [] ? LNetPut+0x122/0x900 [lnet] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870021] [] kmem_cache_alloc_trace+0x132/0x140 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870027] [] ? lnet_md_alloc.isra.2+0x74/0x280 [lnet] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870033] [] LNetPut+0x122/0x900 [lnet] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870056] [] ? cfs_percpt_unlock+0x42/0xd0 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870110] [] ptl_send_buf+0x12f/0x5d0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870117] [] ? LNetMDAttach+0x2db/0x370 [lnet] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870125] [] ? read_tsc+0x9/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870145] [] ptl_send_rpc+0x522/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870165] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870174] [] ? default_spin_lock_flags+0x9/0x10 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870184] [] ? _raw_spin_lock_irqsave+0x2e/0x40 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870205] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870225] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870247] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870268] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870280] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870302] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.870311] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870314] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870320] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870322] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870323] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147697.870325] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.870327] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870327] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.870329] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870330] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.870331] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870335] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870335] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870335] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870335] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870335] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870335] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870338] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147697.870342] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870344] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147697.870348] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870350] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147697.870353] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870355] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.870363] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 788*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61740kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.870369] Node 0 Normal: 1301*4kB (UEM) 4105*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41116kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.870376] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147697.870378] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147697.870379] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147697.870380] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.870381] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.923143] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147697.923148] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147697.923150] 330997 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147697.923150] 4037716 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147697.923155] __slab_alloc: 27 callbacks suppressed Mar 9 22:04:44 lustre-utils01 kernel: [147697.923157] SLUB: Unable to allocate memory on node -1 (gfp=0x40) Mar 9 22:04:44 lustre-utils01 kernel: [147697.923159] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.923161] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.923311] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: out of memory at /var/lib/dkms/lustre-client-modules-2.5.0/2.5.0/build/lnet/include/lnet/lib-lnet.h:506 (tried to alloc '(msg)' = 352) Mar 9 22:04:44 lustre-utils01 kernel: [147697.923578] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) Skipped 13 previous similar messages Mar 9 22:04:44 lustre-utils01 kernel: [147697.923707] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: 716617015 total bytes allocated by lnet Mar 9 22:04:44 lustre-utils01 kernel: [147697.923841] LNetError: 2001:0:(lib-lnet.h:506:lnet_msg_alloc()) Skipped 13 previous similar messages Mar 9 22:04:44 lustre-utils01 kernel: [147697.923971] LNetError: 2001:0:(lib-move.c:2138:LNetPut()) Dropping PUT to 12345-172.17.117.150@tcp: ENOMEM on lnet_msg_t Mar 9 22:04:44 lustre-utils01 kernel: [147697.924112] LNetError: 2001:0:(lib-move.c:2138:LNetPut()) Skipped 12 previous similar messages Mar 9 22:04:44 lustre-utils01 kernel: [147697.924260] LustreError: 2001:0:(niobuf.c:96:ptl_send_buf()) LNetPut(12345-172.17.117.150@tcp, 17, 1461983444039708) failed: -12 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924408] LustreError: 2001:0:(niobuf.c:96:ptl_send_buf()) Skipped 12 previous similar messages Mar 9 22:04:44 lustre-utils01 kernel: [147697.924562] Lustre: 2001:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1394402599/real 1394402599] req@ffff8803dda02800 x1461983444039708/t0(0) o103->lus13-OST001f-osc-ffff880412525000@172.17.117.150@tcp:17/18 lens 344/224 e 0 to 1 dl 1394402829 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924579] Lustre: lus13-OST001f-osc-ffff880412525000: Connection to lus13-OST001f (at 172.17.117.150@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 22:04:44 lustre-utils01 kernel: [147697.924605] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924607] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147697.924609] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147697.924619] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924621] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924632] [] ? console_unlock.part.8+0x22e/0x280 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924636] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924639] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924641] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924645] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924695] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924699] [] ? check_preempt_curr+0x84/0xa0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924701] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924703] [] ? ttwu_do_activate.constprop.86+0x66/0x70 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924725] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924746] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924766] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924788] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924790] [] ? default_wake_function+0x12/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924810] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924831] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924850] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924870] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924891] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924913] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924915] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924936] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.924939] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924941] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924944] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924946] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924947] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147697.924949] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.924951] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924952] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.924953] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924954] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.924956] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924959] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924959] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924959] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924959] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924959] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924959] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924962] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147697.924973] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924976] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147697.924980] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924982] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147697.924986] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.924988] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.924995] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 788*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61740kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.925002] Node 0 Normal: 1301*4kB (UEM) 4105*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41116kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.925009] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147697.925010] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147697.925012] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147697.925013] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.925014] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.977772] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147697.977775] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147697.977776] 330997 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147697.977777] 4037716 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147697.977782] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:44 lustre-utils01 kernel: [147697.977784] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977786] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977912] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977915] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147697.977917] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147697.977926] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977957] [] ? number.isra.2+0x355/0x390 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977960] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977962] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977964] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977987] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147697.977990] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977992] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977994] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.977997] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978044] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978051] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978053] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978075] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978097] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978117] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978138] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978141] [] ? default_wake_function+0x12/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978167] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978190] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978210] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978230] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978251] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978273] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978275] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978296] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147697.978300] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978302] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978305] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978307] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978308] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147697.978309] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.978311] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978312] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.978314] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978315] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147697.978316] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978319] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978319] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978319] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978319] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978319] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978319] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978322] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147697.978326] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978328] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147697.978332] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978334] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147697.978337] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978339] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.978347] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 788*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61740kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.978354] Node 0 Normal: 1301*4kB (UEM) 4105*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41116kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.978360] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147697.978362] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147697.978363] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147697.978364] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147697.978365] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.031037] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147698.031040] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147698.031041] 330997 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.031042] 4037716 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.031048] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:44 lustre-utils01 kernel: [147698.031051] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031086] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031287] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031291] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147698.031293] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147698.031302] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031307] [] ? number.isra.2+0x355/0x390 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031309] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031311] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031313] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031337] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031346] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031351] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031353] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031357] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031417] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031426] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031429] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031457] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031480] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031500] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031521] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031525] [] ? default_wake_function+0x12/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031544] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031570] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031594] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031614] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031636] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031657] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031660] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031681] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.031684] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031686] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031689] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031691] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031693] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147698.031694] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.031696] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031697] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.031698] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031699] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.031700] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031704] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031704] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031704] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031704] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031704] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031704] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031706] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147698.031710] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031713] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.031716] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031718] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.031723] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031725] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.031733] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 788*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61740kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.031739] Node 0 Normal: 1301*4kB (UEM) 4105*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41116kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.031747] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147698.031748] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147698.031750] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147698.031751] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.031751] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.084903] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147698.084907] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147698.084908] 330997 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.084909] 4037716 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.084914] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:44 lustre-utils01 kernel: [147698.084916] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.084918] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085350] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085354] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147698.085355] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147698.085365] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085369] [] ? number.isra.2+0x355/0x390 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085371] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085373] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085375] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085383] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085387] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085388] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085392] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085440] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085453] [] ? LNetPut+0x320/0x900 [lnet] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085455] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085475] [] ? ptl_send_buf+0x12f/0x5d0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085496] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085518] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085537] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085559] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085578] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085599] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085619] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085639] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085660] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085681] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085685] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085706] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.085709] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085712] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085715] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085717] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085718] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147698.085719] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.085721] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085722] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.085723] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085724] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.085763] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085768] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085768] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085768] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085768] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085768] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085768] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085770] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147698.085775] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085777] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.085781] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085783] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.085787] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085788] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.085796] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 787*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61708kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.085802] Node 0 Normal: 1301*4kB (UEM) 4103*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41100kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.085809] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147698.085810] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147698.085812] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147698.085813] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.085813] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.140211] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147698.140216] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147698.140217] 331093 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.140218] 4037720 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.140224] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:44 lustre-utils01 kernel: [147698.140227] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140228] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140492] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140496] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147698.140498] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147698.140509] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140514] [] ? number.isra.2+0x355/0x390 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140517] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140521] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140523] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140550] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140556] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140562] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140564] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140567] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140620] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140628] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140631] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140660] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140683] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140708] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140736] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140744] [] ? default_wake_function+0x12/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140766] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140793] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140817] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140846] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140870] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140897] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140903] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140932] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.140938] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140942] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140946] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140948] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140950] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147698.140951] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.140953] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140954] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.140956] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140956] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.140958] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140963] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140963] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140963] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140963] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140963] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140963] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140967] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147698.140973] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140975] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.140980] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140982] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.140987] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.140991] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.140999] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 787*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61708kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.141008] Node 0 Normal: 1301*4kB (UEM) 4103*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41100kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.141020] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147698.141022] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147698.141040] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147698.141042] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.141043] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.199435] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147698.199438] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147698.199439] 331085 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.199440] 4037720 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.199446] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:44 lustre-utils01 kernel: [147698.199448] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199450] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199555] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199558] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147698.199559] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147698.199569] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199573] [] ? number.isra.2+0x355/0x390 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199575] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199577] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199579] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199601] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199604] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199607] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199609] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199612] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199659] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199666] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199669] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199691] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199712] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199732] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199754] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199757] [] ? default_wake_function+0x12/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199777] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199799] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199818] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199838] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199859] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199880] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199883] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199904] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.199907] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199911] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199916] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199919] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199921] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147698.199923] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.199925] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199926] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.199927] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199928] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.199929] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199933] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199933] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199933] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199933] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199933] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199933] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199936] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147698.199940] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199942] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.199947] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199949] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.199953] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199956] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.199966] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 787*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61708kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.199975] Node 0 Normal: 1301*4kB (UEM) 4103*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41100kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.199982] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147698.199984] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147698.199985] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147698.199986] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.199987] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.254367] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147698.254371] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147698.254372] 331085 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.254372] 4037720 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.254378] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:44 lustre-utils01 kernel: [147698.254380] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254381] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254472] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254475] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147698.254476] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147698.254486] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254490] [] ? number.isra.2+0x355/0x390 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254492] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254494] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254496] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254518] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254521] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254524] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254526] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254529] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254576] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254583] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254585] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254607] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254628] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254648] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254670] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254673] [] ? default_wake_function+0x12/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254693] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254714] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254734] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254753] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254774] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254796] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254813] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254844] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.254849] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254851] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254855] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254857] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254858] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147698.254860] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.254862] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254863] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.254864] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254865] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.254867] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254870] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254870] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254870] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254870] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254870] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254870] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254872] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147698.254876] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254879] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.254882] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254884] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.254888] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254890] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.254897] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 787*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61708kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.254904] Node 0 Normal: 1301*4kB (UEM) 4103*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41100kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.254911] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147698.254912] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147698.254914] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147698.254915] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.254915] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.308450] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147698.308454] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147698.308455] 331085 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.308456] 4037720 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.308461] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:44 lustre-utils01 kernel: [147698.308463] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308465] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308584] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308587] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147698.308588] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147698.308598] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308602] [] ? number.isra.2+0x355/0x390 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308604] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308606] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308608] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308632] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308635] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308639] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308641] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308645] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308691] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308698] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308701] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308723] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308744] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308766] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308794] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308801] [] ? default_wake_function+0x12/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308820] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308842] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308862] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308882] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308905] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308927] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308930] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308951] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.308954] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308957] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308959] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308962] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308963] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147698.308964] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.308966] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308967] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.308969] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308969] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.308971] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308974] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308974] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308974] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308974] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308974] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308974] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308977] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147698.308980] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308983] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.308987] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308988] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.308992] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.308994] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.309003] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 787*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61708kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.309010] Node 0 Normal: 1301*4kB (UEM) 4103*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41100kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.309017] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147698.309019] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147698.309020] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:44 lustre-utils01 kernel: [147698.309021] Free swap = 38730772kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.309022] Total swap = 38793204kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.362292] 4194288 pages RAM Mar 9 22:04:44 lustre-utils01 kernel: [147698.362296] 85481 pages reserved Mar 9 22:04:44 lustre-utils01 kernel: [147698.362297] 331085 pages shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.362297] 4037720 pages non-shared Mar 9 22:04:44 lustre-utils01 kernel: [147698.362303] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:44 lustre-utils01 kernel: [147698.362305] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362307] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362407] ptlrpcd_1: page allocation failure: order:0, mode:0x200040 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362410] Pid: 2001, comm: ptlrpcd_1 Tainted: GF O 3.8.0-37-generic #53~precise1-Ubuntu Mar 9 22:04:44 lustre-utils01 kernel: [147698.362412] Call Trace: Mar 9 22:04:44 lustre-utils01 kernel: [147698.362421] [] warn_alloc_failed+0xf6/0x150 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362426] [] ? number.isra.2+0x355/0x390 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362427] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362430] [] __alloc_pages_nodemask+0x860/0x990 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362431] [] ? put_dec+0x72/0x90 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362454] [] ? libcfs_debug_vmsg2+0x52d/0xb10 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362458] [] alloc_pages_current+0xba/0x170 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362460] [] allocate_slab+0x190/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362462] [] new_slab+0x30/0x1a0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362465] [] __slab_alloc+0x17d/0x2b2 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362523] [] ? null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362532] [] ? cfs_trace_unlock_tcd+0x31/0x90 [libcfs] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362534] [] __kmalloc+0x162/0x190 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362556] [] null_alloc_repbuf+0x66/0x3f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362578] [] sptlrpc_cli_alloc_repbuf+0x13f/0x210 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362598] [] ptl_send_rpc+0x884/0xc20 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362619] [] ? sptlrpc_req_refresh_ctx+0x183/0x990 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362623] [] ? default_wake_function+0x12/0x20 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362643] [] ptlrpc_send_new_req+0x1f2/0x7c0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362664] [] ? ptlrpc_fail_import+0x9d/0x2a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362684] [] ptlrpc_check_set+0x73d/0x20f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362704] [] ? __ptlrpc_req_finished+0x4d5/0x6f0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362725] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362746] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362748] [] ? try_to_wake_up+0x200/0x200 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362770] [] ? ptlrpcd_check+0x5a0/0x5a0 [ptlrpc] Mar 9 22:04:44 lustre-utils01 kernel: [147698.362773] [] kthread+0xc0/0xd0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362775] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362778] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362780] [] ? flush_kthread_worker+0xb0/0xb0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362781] Mem-Info: Mar 9 22:04:44 lustre-utils01 kernel: [147698.362783] Node 0 DMA per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.362785] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362786] Node 0 DMA32 per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.362787] CPU 0: hi: 186, btch: 31 usd: 75 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362788] Node 0 Normal per-cpu: Mar 9 22:04:44 lustre-utils01 kernel: [147698.362789] CPU 0: hi: 186, btch: 31 usd: 155 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362793] active_anon:44530 inactive_anon:5943 isolated_anon:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362793] active_file:66833 inactive_file:18730 isolated_file:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362793] unevictable:2390 dirty:56 writeback:0 unstable:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362793] free:29696 slab_reclaimable:72123 slab_unreclaimable:3663530 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362793] mapped:19540 shmem:33 pagetables:1408 bounce:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362793] free_cma:0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362795] Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 9 22:04:44 lustre-utils01 kernel: [147698.362799] lowmem_reserve[]: 0 3008 16112 16112 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362802] Node 0 DMA32 free:61740kB min:12604kB low:15752kB high:18904kB active_anon:53156kB inactive_anon:880kB active_file:45044kB inactive_file:6072kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3080384kB managed:3031732kB mlocked:0kB dirty:16kB writeback:0kB mapped:31668kB shmem:0kB slab_reclaimable:18940kB slab_unreclaimable:2863788kB kernel_stack:24kB pagetables:280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.362806] lowmem_reserve[]: 0 0 13104 13104 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362808] Node 0 Normal free:41140kB min:54912kB low:68640kB high:82368kB active_anon:124964kB inactive_anon:22892kB active_file:222288kB inactive_file:68848kB unevictable:9560kB isolated(anon):0kB isolated(file):0kB present:13418496kB managed:13368476kB mlocked:9560kB dirty:208kB writeback:0kB mapped:46492kB shmem:132kB slab_reclaimable:269552kB slab_unreclaimable:11790328kB kernel_stack:2240kB pagetables:5352kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 22:04:44 lustre-utils01 kernel: [147698.362811] lowmem_reserve[]: 0 0 0 0 Mar 9 22:04:44 lustre-utils01 kernel: [147698.362813] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (UM) = 15904kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.362821] Node 0 DMA32: 1*4kB (E) 1*8kB (M) 994*16kB (UM) 787*32kB (UM) 258*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61708kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.362827] Node 0 Normal: 1301*4kB (UEM) 4103*8kB (M) 0*16kB 8*32kB (R) 4*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 41100kB Mar 9 22:04:44 lustre-utils01 kernel: [147698.362834] 87583 total pagecache pages Mar 9 22:04:44 lustre-utils01 kernel: [147698.362836] 1139 pages in swap cache Mar 9 22:04:44 lustre-utils01 kernel: [147698.362837] Swap cache stats: add 21546, delete 20407, find 34990/35620 Mar 9 22:04:45 lustre-utils01 kernel: [147698.362838] Free swap = 38730772kB Mar 9 22:04:45 lustre-utils01 kernel: [147698.362839] Total swap = 38793204kB Mar 9 22:04:45 lustre-utils01 kernel: [147698.415991] 4194288 pages RAM Mar 9 22:04:45 lustre-utils01 kernel: [147698.415995] 85481 pages reserved Mar 9 22:04:45 lustre-utils01 kernel: [147698.415996] 331085 pages shared Mar 9 22:04:45 lustre-utils01 kernel: [147698.415997] 4037720 pages non-shared Mar 9 22:04:45 lustre-utils01 kernel: [147698.416002] SLUB: Unable to allocate memory on node -1 (gfp=0x8040) Mar 9 22:04:45 lustre-utils01 kernel: [147698.416004] cache: kmalloc-512, object size: 512, buffer size: 512, default order: 0, min order: 0 Mar 9 22:04:45 lustre-utils01 kernel: [147698.416006] node 0: slabs: 38930, objs: 311440, free: 0 Mar 9 22:04:45 lustre-utils01 kernel: [147698.417528] LustreError: 2001:0:(osc_lock.c:597:osc_lock_upcall()) LBUG Mar 9 22:04:45 lustre-utils01 kernel: [147698.417681] Pid: 2001, comm: ptlrpcd_1 Mar 9 22:04:45 lustre-utils01 kernel: [147698.417682] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417682] Call Trace: Mar 9 22:04:45 lustre-utils01 kernel: [147698.417705] [] libcfs_debug_dumpstack+0x57/0x80 [libcfs] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417712] [] lbug_with_loc+0x47/0xc0 [libcfs] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417823] [] ? cl_env_nested_get+0xb0/0xc0 [obdclass] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417844] [] osc_lock_upcall+0x41a/0x5f0 [osc] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417851] [] ? osc_lock_upcall+0x0/0x5f0 [osc] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417856] [] osc_enqueue_fini+0xf9/0x240 [osc] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417861] [] osc_enqueue_interpret+0xe2/0x1f0 [osc] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417900] [] ptlrpc_check_set+0x3df/0x20f0 [ptlrpc] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417923] [] ptlrpcd_check+0x36f/0x5a0 [ptlrpc] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417944] [] ptlrpcd+0x22b/0x3a0 [ptlrpc] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417949] [] ? default_wake_function+0x0/0x20 Mar 9 22:04:45 lustre-utils01 kernel: [147698.417971] [] ? ptlrpcd+0x0/0x3a0 [ptlrpc] Mar 9 22:04:45 lustre-utils01 kernel: [147698.417974] [] kthread+0xc0/0xd0 Mar 9 22:04:45 lustre-utils01 kernel: [147698.417976] [] ? kthread+0x0/0xd0 Mar 9 22:04:45 lustre-utils01 kernel: [147698.417979] [] ret_from_fork+0x7c/0xb0 Mar 9 22:04:45 lustre-utils01 kernel: [147698.417981] [] ? kthread+0x0/0xd0 Mar 9 22:04:45 lustre-utils01 kernel: [147698.417982] Mar 9 22:04:45 lustre-utils01 kernel: [147700.029968] Lustre: lus13-MDT0000-mdc-ffff880412525000: Connection restored to lus13-MDT0000 (at 172.17.117.153@tcp) Mar 9 22:04:45 lustre-utils01 kernel: [147700.099509] LNetError: 2000:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: out of memory at /var/lib/dkms/lustre-client-modules-2.5.0/2.5.0/build/lnet/include/lnet/lib-lnet.h:506 (tried to alloc '(msg)' = 352) Mar 9 22:04:45 lustre-utils01 kernel: [147700.099848] LNetError: 2000:0:(lib-lnet.h:506:lnet_msg_alloc()) LNET: 717884943 total bytes allocated by lnet Mar 9 22:04:45 lustre-utils01 kernel: [147700.100054] LNetError: 2000:0:(lib-move.c:2138:LNetPut()) Dropping PUT to 12345-172.17.117.153@tcp: ENOMEM on lnet_msg_t Mar 9 22:04:45 lustre-utils01 kernel: [147700.100260] LustreError: 2000:0:(niobuf.c:96:ptl_send_buf()) LNetPut(12345-172.17.117.153@tcp, 17, 1461983444077436) failed: -12 Mar 9 22:04:45 lustre-utils01 kernel: [147700.212755] LustreError: 24586:0:(ldlm_request.c:1908:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -12 Mar 9 22:04:45 lustre-utils01 kernel: [147700.309100] LustreError: 26417:0:(client.c:719:__ptlrpc_request_alloc()) request allocation out of memory Mar 9 22:04:45 lustre-utils01 kernel: [147700.309262] LustreError: 26417:0:(ldlm_request.c:1908:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -12 Mar 9 22:04:45 lustre-utils01 kernel: [147700.309397] LustreError: 26417:0:(ldlm_request.c:1908:ldlm_cli_cancel_list()) Skipped 1 previous similar message Mar 9 22:04:45 lustre-utils01 kernel: [147748.690931] Lustre: lus13-OST0027-osc-ffff880412525000: Connection restored to lus13-OST0027 (at 172.17.117.151@tcp) Mar 9 22:04:45 lustre-utils01 kernel: [147748.690935] Lustre: Skipped 1 previous similar message Mar 9 22:04:45 lustre-utils01 kernel: [147781.059253] Lustre: lus13-OST000f-osc-ffff880412525000: Connection restored to lus13-OST000f (at 172.17.117.147@tcp) Mar 9 22:04:45 lustre-utils01 kernel: [147781.059258] Lustre: Skipped 11 previous similar messages Mar 9 22:04:53 lustre-utils01 kernel: [147791.511192] LustreError: 11-0: lus13-OST0009-osc-ffff880412525000: Communicating with 172.17.117.145@tcp, operation ost_connect failed with -16. Mar 9 22:04:59 lustre-utils01 kernel: [147797.507066] LustreError: 11-0: lus13-OST0006-osc-ffff880412525000: Communicating with 172.17.117.145@tcp, operation ost_connect failed with -16. Mar 9 22:04:59 lustre-utils01 kernel: [147797.507251] LustreError: Skipped 1 previous similar message Mar 9 22:05:16 lustre-utils01 kernel: [147814.720446] LustreError: 11-0: lus13-OST0012-osc-ffff880412525000: Communicating with 172.17.117.147@tcp, operation ost_connect failed with -16. Mar 9 22:05:21 lustre-utils01 kernel: [147820.216190] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=15002 jiffies, g=50154932, c=50154931, q=2024406) Mar 9 22:05:21 lustre-utils01 kernel: [147820.216428] INFO: Stall ended before state dump start Mar 9 22:05:51 lustre-utils01 kernel: [147849.365777] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1394402702/real 1394402750] req@ffff880236988000 x1461983445991768/t0(0) o103->lus13-OST000e-osc-ffff880412525000@172.17.117.146@tcp:17/18 lens 352/224 e 0 to 1 dl 1394402932 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 22:05:51 lustre-utils01 kernel: [147849.365784] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 54 previous similar messages Mar 9 22:05:51 lustre-utils01 kernel: [147849.365882] Lustre: lus13-OST000e-osc-ffff880412525000: Connection to lus13-OST000e (at 172.17.117.146@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 22:05:51 lustre-utils01 kernel: [147849.365884] Lustre: Skipped 41 previous similar messages Mar 9 22:05:51 lustre-utils01 kernel: [147849.366910] LustreError: 166-1: MGC172.17.31.16@tcp: Connection to MGS (at 172.17.31.16@tcp) was lost; in progress operations using this service will fail Mar 9 22:05:53 lustre-utils01 kernel: [147850.984740] LustreError: 166-1: MGC172.17.117.152@tcp: Connection to MGS (at 172.17.117.152@tcp) was lost; in progress operations using this service will fail Mar 9 22:05:59 lustre-utils01 kernel: [147858.246572] LNetError: 1992:0:(socklnd_cb.c:1725:ksocknal_recv_hello()) Error -104 reading HELLO from 172.17.31.56 Mar 9 22:05:59 lustre-utils01 kernel: [147858.246790] LNetError: 11b-b: Connection to 172.17.31.56@tcp at host 172.17.31.56 on port 988 was reset: is it running a compatible version of Lustre and is 172.17.31.56@tcp one of its NIDs? Mar 9 22:06:00 lustre-utils01 kernel: [147858.852240] LNetError: 2026:0:(socklnd_cb.c:1725:ksocknal_recv_hello()) Error -104 reading HELLO from 172.17.149.8 Mar 9 22:06:00 lustre-utils01 kernel: [147858.852471] LNetError: 11b-b: Connection to 172.17.149.8@tcp at host 172.17.149.8 on port 988 was reset: is it running a compatible version of Lustre and is 172.17.149.8@tcp one of its NIDs? Mar 9 22:06:01 lustre-utils01 kernel: [147859.423212] Lustre: lus06-OST000f-osc-ffff880403c51800: Connection to lus06-OST000f (at 172.17.149.8@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 22:06:01 lustre-utils01 kernel: [147859.423218] Lustre: Skipped 198 previous similar messages Mar 9 22:06:02 lustre-utils01 kernel: [147860.622801] LNetError: 2064:0:(socklnd_cb.c:1725:ksocknal_recv_hello()) Error -104 reading HELLO from 172.17.31.54 Mar 9 22:06:02 lustre-utils01 kernel: [147860.623046] LNetError: 11b-b: Connection to 172.17.31.54@tcp at host 172.17.31.54 on port 988 was reset: is it running a compatible version of Lustre and is 172.17.31.54@tcp one of its NIDs? Mar 9 22:06:02 lustre-utils01 kernel: [147860.633451] Lustre: lus08-OST0001-osc-ffff8804282d9400: Connection restored to lus08-OST0001 (at 172.17.31.18@tcp) Mar 9 22:06:02 lustre-utils01 kernel: [147860.633456] Lustre: Skipped 13 previous similar messages Mar 9 22:06:02 lustre-utils01 kernel: [147860.655461] LustreError: 11-0: lus05-MDT0000-mdc-ffff88040db36c00: Communicating with 172.17.148.134@tcp, operation mds_connect failed with -16. Mar 9 22:07:06 lustre-utils01 kernel: [147925.204141] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394402721/real 0] req@ffff8802fb3c3c00 x1461983446024264/t0(0) o8->lus13-OST0015-osc-ffff880412525000@172.17.117.148@tcp:28/4 lens 400/544 e 0 to 1 dl 1394402826 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 9 22:07:06 lustre-utils01 kernel: [147925.204146] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 271 previous similar messages Mar 9 22:07:51 lustre-utils01 kernel: [147969.736344] Lustre: lus13-OST0021-osc-ffff880412525000: Connection to lus13-OST0021 (at 172.17.117.150@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 22:07:51 lustre-utils01 kernel: [147969.736351] Lustre: Skipped 3 previous similar messages Mar 9 22:08:08 lustre-utils01 kernel: [147986.398131] LustreError: 167-0: lus09-OST000f-osc-ffff8804088ffc00: This client was evicted by lus09-OST000f; in progress operations using this service will fail. Mar 9 22:08:08 lustre-utils01 kernel: [147986.398356] LustreError: Skipped 7 previous similar messages Mar 9 22:08:11 lustre-utils01 kernel: [147989.769855] Lustre: lus09-OST000f-osc-ffff8804088ffc00: Connection restored to lus09-OST000f (at 172.17.31.54@tcp) Mar 9 22:08:11 lustre-utils01 kernel: [147989.769860] Lustre: Skipped 196 previous similar messages Mar 9 22:08:19 lustre-utils01 kernel: [147996.244394] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394402663/real 0] req@ffff880304820000 x1461983445665868/t0(0) o103->lus13-OST0027-osc-ffff880412525000@172.17.117.150@tcp:17/18 lens 352/224 e 0 to 1 dl 1394402893 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 9 22:08:30 lustre-utils01 kernel: [147996.244400] Lustre: 2000:0:(client.c:1897:ptlrpc_expire_one_request()) Skipped 24098 previous similar messages Mar 9 22:08:30 lustre-utils01 kernel: [148004.958710] Lustre: lus13-OST001e-osc-ffff880412525000: Connection to lus13-OST001e (at 172.17.117.150@tcp) was lost; in progress operations using this service will wait for recovery to complete Mar 9 22:08:30 lustre-utils01 kernel: [148004.958715] Lustre: Skipped 3 previous similar messages Mar 10 09:26:55 lustre-utils01 kernel: imklog 5.8.6, log source = /proc/kmsg started. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Initializing cgroup subsys cpuset Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Initializing cgroup subsys cpu Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Linux version 3.8.0-37-generic (buildd@orlo) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #53~precise1-Ubuntu SMP Wed Feb 19 21:37:54 UTC 2014 (Ubuntu 3.8.0-37.53~precise1-generic 3.8.13.18) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.8.0-37-generic root=/dev/mapper/precise--dev64-root ro consoleblank=0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] KERNEL supported cpus: Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Intel GenuineIntel Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] AMD AuthenticAMD Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Centaur CentaurHauls Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Disabled fast string operations Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] e820: BIOS-provided physical RAM map: Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000000ca000-0x00000000000cbfff] reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000000dc000-0x00000000000fffff] reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bfeeffff] usable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000bfef0000-0x00000000bfefefff] ACPI data Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000bfeff000-0x00000000bfefffff] ACPI NVS Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000bff00000-0x00000000bfffffff] usable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x00000000fffe0000-0x00000000ffffffff] reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000043fffffff] usable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] NX (Execute Disable) protection: active Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] SMBIOS 2.4 present. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] DMI: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Hypervisor detected: VMware Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] No AGP bridge found Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] e820: last_pfn = 0x440000 max_arch_pfn = 0x400000000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] MTRR default type: uncachable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] MTRR fixed ranges enabled: Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 00000-9FFFF write-back Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] A0000-BFFFF uncachable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] C0000-CBFFF write-protect Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] CC000-EFFFF uncachable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] F0000-FFFFF write-protect Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] MTRR variable ranges enabled: Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 0 base 00C0000000 mask FFC0000000 uncachable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 1 base 0000000000 mask FC00000000 write-back Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 2 base 0400000000 mask FFC0000000 write-back Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 3 disabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 4 disabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 5 disabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 6 disabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] 7 disabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] x86 PAT enabled: cpu 0, old 0x0, new 0x7010600070106 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] original variable MTRRs Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] reg 0, base: 3GB, range: 1GB, type UC Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] reg 1, base: 0GB, range: 16GB, type WB Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] reg 2, base: 16GB, range: 1GB, type WB Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] total RAM covered: 16384M Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Found optimal setting for mtrr clean up Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 5 lose cover RAM: 0G Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] New variable MTRRs Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] reg 1, base: 2GB, range: 1GB, type WB Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] reg 2, base: 4GB, range: 4GB, type WB Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] reg 3, base: 8GB, range: 8GB, type WB Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] reg 4, base: 16GB, range: 1GB, type WB Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] x86 PAT enabled: cpu 0, old 0x0, new 0x7010600070106 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] e820: update [mem 0xc0000000-0xffffffff] usable ==> reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] e820: last_pfn = 0xc0000 max_arch_pfn = 0x400000000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] found SMP MP-table at [mem 0x000f6bf0-0x000f6bff] mapped at [ffff8800000f6bf0] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] initial memory mapped: [mem 0x00000000-0x1fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] init_memory_mapping: [mem 0x00000000-0xbfffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] [mem 0x00000000-0xbfffffff] page 2M Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] kernel direct mapping tables up to 0xbfffffff @ [mem 0x1fffc000-0x1fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] init_memory_mapping: [mem 0x100000000-0x43fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] [mem 0x100000000-0x43fffffff] page 2M Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] kernel direct mapping tables up to 0x43fffffff @ [mem 0xbfff2000-0xbfffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] RAMDISK: [mem 0x35ff4000-0x36ff1fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: RSDP 00000000000f6b80 00024 (v02 PTLTD ) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: XSDT 00000000bfef00ff 0005C (v01 INTEL 440BX 06040000 VMW 01324272) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: FACP 00000000bfefee98 000F4 (v04 INTEL 440BX 06040000 PTL 000F4240) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: DSDT 00000000bfef0367 0EB31 (v01 PTLTD Custom 06040000 MSFT 03000001) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: FACS 00000000bfefffc0 00040 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: BOOT 00000000bfef033f 00028 (v01 PTLTD $SBFTBL$ 06040000 LTP 00000001) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: APIC 00000000bfef02ef 00050 (v01 PTLTD ? APIC 06040000 LTP 00000000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: MCFG 00000000bfef02b3 0003C (v01 PTLTD $PCITBL$ 06040000 LTP 00000001) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: SRAT 00000000bfef01fb 000B8 (v02 VMWARE MEMPLUG 06040000 VMW 00000001) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: HPET 00000000bfef01c3 00038 (v01 VMWARE VMW HPET 06040000 VMW 00000001) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: WAET 00000000bfef019b 00028 (v01 VMWARE VMW WAET 06040000 VMW 00000001) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: Local APIC address 0xfee00000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x43fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xbfffffff] -> [mem 0x00000000-0xbfffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x43fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] NODE_DATA [mem 0x43fffb000-0x43fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] [ffffea0000000000-ffffea0010ffffff] PMD -> [ffff88042f600000-ffff88043f5fffff] on node 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Zone ranges: Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] DMA [mem 0x00010000-0x00ffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Normal [mem 0x100000000-0x43fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Movable zone start for each node Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Early memory node ranges Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] node 0: [mem 0x00010000-0x0009efff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] node 0: [mem 0x00100000-0xbfeeffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] node 0: [mem 0xbff00000-0xbfffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] node 0: [mem 0x100000000-0x43fffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] On node 0 totalpages: 4194175 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] DMA zone: 64 pages used for memmap Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] DMA zone: 6 pages reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] DMA zone: 3913 pages, LIFO batch:0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] DMA32 zone: 12224 pages used for memmap Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] DMA32 zone: 770096 pages, LIFO batch:31 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Normal zone: 53248 pages used for memmap Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Normal zone: 3354624 pages, LIFO batch:31 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: PM-Timer IO Port: 0x1008 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: Local APIC address 0xfee00000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: IRQ0 used by override. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: IRQ2 used by override. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: IRQ9 used by override. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Using ACPI (MADT) for SMP configuration information Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] ACPI: HPET id: 0x8086af01 base: 0xfed00000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] nr_irqs_gsi: 40 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000ca000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000000ca000 - 00000000000cc000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000000cc000 - 00000000000dc000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000000dc000 - 0000000000100000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000bfef0000 - 00000000bfeff000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000bfeff000 - 00000000bff00000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000c0000000 - 00000000e0000000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec10000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fee00000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000fffe0000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PM: Registered nosave memory: 00000000fffe0000 - 0000000100000000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] e820: [mem 0xc0000000-0xdfffffff] available for PCI devices Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Booting paravirtualized kernel on bare hardware Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:1 nr_node_ids:1 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff88043fc00000 s84928 r8192 d21568 u2097152 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] pcpu-alloc: s84928 r8192 d21568 u2097152 alloc=1*2097152 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] pcpu-alloc: [0] 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 4128633 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Policy zone: Normal Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.8.0-37-generic root=/dev/mapper/precise--dev64-root ro consoleblank=0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] __ex_table already sorted, skipping sort Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Checking aperture... Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] No AGP bridge found Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Calgary: detecting Calgary via BIOS EBDA area Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing! Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Memory: 16416116k/17825792k available (7180k kernel code, 1049092k absent, 360584k reserved, 6063k data, 1016k init) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Hierarchical RCU implementation. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] NR_IRQS:16640 nr_irqs:256 16 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Extended CMOS year: 2000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] Console: colour VGA+ 80x25 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] console [tty0] enabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] allocated 67108864 bytes of page_cgroup Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000000] hpet clockevent registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.004000] TSC freq read from hypervisor : 2533.423 MHz Mar 10 09:26:55 lustre-utils01 kernel: [ 0.004000] tsc: Detected 2533.423 MHz processor Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000002] Calibrating delay loop (skipped) preset value.. 5066.84 BogoMIPS (lpj=10133692) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000140] pid_max: default: 32768 minimum: 301 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000238] Security Framework initialized Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000338] AppArmor: AppArmor initialized Mar 10 09:26:55 lustre-utils01 kernel: [ 0.000406] Yama: becoming mindful. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.001969] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.009069] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.011883] Mount-cache hash table entries: 256 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.012153] Initializing cgroup subsys cpuacct Mar 10 09:26:55 lustre-utils01 kernel: [ 0.012228] Initializing cgroup subsys memory Mar 10 09:26:55 lustre-utils01 kernel: [ 0.012303] Initializing cgroup subsys devices Mar 10 09:26:55 lustre-utils01 kernel: [ 0.012373] Initializing cgroup subsys freezer Mar 10 09:26:55 lustre-utils01 kernel: [ 0.012491] Initializing cgroup subsys blkio Mar 10 09:26:55 lustre-utils01 kernel: [ 0.013455] Initializing cgroup subsys perf_event Mar 10 09:26:55 lustre-utils01 kernel: [ 0.013544] Initializing cgroup subsys hugetlb Mar 10 09:26:55 lustre-utils01 kernel: [ 0.013671] Disabled fast string operations Mar 10 09:26:55 lustre-utils01 kernel: [ 0.013744] CPU: Physical Processor ID: 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.013821] mce: CPU supports 0 MCE banks Mar 10 09:26:55 lustre-utils01 kernel: [ 0.014036] Last level iTLB entries: 4KB 512, 2MB 7, 4MB 7 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.014036] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.014036] tlb_flushall_shift: 6 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.025496] Freeing SMP alternatives: 24k freed Mar 10 09:26:55 lustre-utils01 kernel: [ 0.031505] ACPI: Core revision 20121018 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.046037] ftrace: allocating 29409 entries in 115 pages Mar 10 09:26:55 lustre-utils01 kernel: [ 0.082405] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.122212] smpboot: CPU0: Intel(R) Xeon(R) CPU E5540 @ 2.53GHz (fam: 06, model: 1a, stepping: 04) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.227757] Performance Events: 16-deep LBR, Nehalem events, core PMU driver. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.227941] perf_event_intel: CPU erratum AAJ80 worked around Mar 10 09:26:55 lustre-utils01 kernel: [ 0.228032] perf_event_intel: CPUID marked event: 'cpu cycles' unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.228124] perf_event_intel: CPUID marked event: 'instructions' unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.228235] perf_event_intel: CPUID marked event: 'bus cycles' unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.228327] perf_event_intel: CPUID marked event: 'cache references' unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.230705] perf_event_intel: CPUID marked event: 'cache misses' unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.230800] perf_event_intel: CPUID marked event: 'branch instructions' unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 0.230920] ... version: 1 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.230988] ... bit width: 48 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.231056] ... generic registers: 4 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.231124] ... value mask: 0000ffffffffffff Mar 10 09:26:55 lustre-utils01 kernel: [ 0.231211] ... max period: 000000007fffffff Mar 10 09:26:55 lustre-utils01 kernel: [ 0.231298] ... fixed-purpose events: 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.231365] ... event mask: 000000000000000f Mar 10 09:26:55 lustre-utils01 kernel: [ 0.232387] Brought up 1 CPUs Mar 10 09:26:55 lustre-utils01 kernel: [ 0.232453] smpboot: Total of 1 processors activated (5066.84 BogoMIPS) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.232697] x86 PAT enabled: cpu 0, old 0x0, new 0x7010600070106 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.232839] NMI watchdog: disabled (cpu0): hardware events not enabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.233020] devtmpfs: initialized Mar 10 09:26:55 lustre-utils01 kernel: [ 0.234734] EVM: security.selinux Mar 10 09:26:55 lustre-utils01 kernel: [ 0.234801] EVM: security.SMACK64 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.234862] EVM: security.capability Mar 10 09:26:55 lustre-utils01 kernel: [ 0.234970] PM: Registering ACPI NVS region [mem 0xbfeff000-0xbfefffff] (4096 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.235857] regulator-dummy: no parameters Mar 10 09:26:55 lustre-utils01 kernel: [ 0.235984] NET: Registered protocol family 16 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.236183] ACPI: bus type pci registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.236493] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.236631] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.258317] PCI: Using configuration type 1 for base access Mar 10 09:26:55 lustre-utils01 kernel: [ 0.259837] bio: create slab at 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.260052] ACPI: Added _OSI(Module Device) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.260152] ACPI: Added _OSI(Processor Device) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.260810] ACPI: Added _OSI(3.0 _SCP Extensions) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.260913] ACPI: Added _OSI(Processor Aggregator Device) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.264232] ACPI: EC: Look up EC in DSDT Mar 10 09:26:55 lustre-utils01 kernel: [ 0.270794] [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored Mar 10 09:26:55 lustre-utils01 kernel: [ 0.272906] ACPI: Interpreter enabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.273760] ACPI: (supports S0 S1 S4 S5) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.273993] ACPI: Using IOAPIC for interrupt routing Mar 10 09:26:55 lustre-utils01 kernel: [ 0.322451] ACPI: No dock devices found. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.322537] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug Mar 10 09:26:55 lustre-utils01 kernel: [ 0.323386] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.323497] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.325202] PCI host bridge to bus 0000:00 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.325297] pci_bus 0000:00: root bus resource [bus 00-ff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.325383] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.325480] pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.325597] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.325695] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.325795] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.325903] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.326000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.326091] pci_bus 0000:00: root bus resource [io 0x0d00-0xfeff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.326269] pci 0000:00:00.0: [8086:7190] type 00 class 0x060000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.326700] pci 0000:00:01.0: [8086:7191] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.327062] pci 0000:00:07.0: [8086:7110] type 00 class 0x060100 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.327680] pci 0000:00:07.1: [8086:7111] type 00 class 0x01018a Mar 10 09:26:55 lustre-utils01 kernel: [ 0.329565] pci 0000:00:07.1: reg 20: [io 0x10c0-0x10cf] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.330298] pci 0000:00:07.3: [8086:7113] type 00 class 0x068000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.332107] pci 0000:00:07.3: quirk: [io 0x1000-0x103f] claimed by PIIX4 ACPI Mar 10 09:26:55 lustre-utils01 kernel: [ 0.332246] pci 0000:00:07.3: quirk: [io 0x1040-0x104f] claimed by PIIX4 SMB Mar 10 09:26:55 lustre-utils01 kernel: [ 0.332569] pci 0000:00:07.7: [15ad:0740] type 00 class 0x088000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.333206] pci 0000:00:07.7: reg 10: [io 0x1080-0x10bf] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.333769] pci 0000:00:07.7: reg 14: [mem 0xd0000000-0xd0001fff 64bit] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.336359] pci 0000:00:0f.0: [15ad:0405] type 00 class 0x030000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.337952] pci 0000:00:0f.0: reg 10: [io 0x10d0-0x10df] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.339378] pci 0000:00:0f.0: reg 14: [mem 0xd8000000-0xdbffffff pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.340762] pci 0000:00:0f.0: reg 18: [mem 0xd0800000-0xd0ffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.348611] pci 0000:00:0f.0: reg 30: [mem 0x00000000-0x00007fff pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.348919] pci 0000:00:11.0: [15ad:0790] type 01 class 0x060401 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.349536] pci 0000:00:15.0: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.350215] pci 0000:00:15.0: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.350459] pci 0000:00:15.1: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.351088] pci 0000:00:15.1: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.351316] pci 0000:00:15.2: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.351943] pci 0000:00:15.2: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.352173] pci 0000:00:15.3: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.352792] pci 0000:00:15.3: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.352999] pci 0000:00:15.4: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.353635] pci 0000:00:15.4: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.353842] pci 0000:00:15.5: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.354470] pci 0000:00:15.5: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.354704] pci 0000:00:15.6: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.355326] pci 0000:00:15.6: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.355558] pci 0000:00:15.7: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.356195] pci 0000:00:15.7: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.356412] pci 0000:00:16.0: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.357028] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.357257] pci 0000:00:16.1: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.357880] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.358093] pci 0000:00:16.2: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.358707] pci 0000:00:16.2: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.358923] pci 0000:00:16.3: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.359623] pci 0000:00:16.3: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.359859] pci 0000:00:16.4: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.360503] pci 0000:00:16.4: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.360712] pci 0000:00:16.5: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.361411] pci 0000:00:16.5: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.361695] pci 0000:00:16.6: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.362378] pci 0000:00:16.6: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.362588] pci 0000:00:16.7: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.363215] pci 0000:00:16.7: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.363437] pci 0000:00:17.0: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.364059] pci 0000:00:17.0: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.364289] pci 0000:00:17.1: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.364891] pci 0000:00:17.1: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.365111] pci 0000:00:17.2: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.365760] pci 0000:00:17.2: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.365983] pci 0000:00:17.3: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.366612] pci 0000:00:17.3: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.366829] pci 0000:00:17.4: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.367454] pci 0000:00:17.4: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.367673] pci 0000:00:17.5: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.368292] pci 0000:00:17.5: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.368501] pci 0000:00:17.6: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.369127] pci 0000:00:17.6: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.369366] pci 0000:00:17.7: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.370005] pci 0000:00:17.7: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.370249] pci 0000:00:18.0: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.370853] pci 0000:00:18.0: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.371062] pci 0000:00:18.1: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.371716] pci 0000:00:18.1: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.371932] pci 0000:00:18.2: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.372551] pci 0000:00:18.2: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.372760] pci 0000:00:18.3: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.373383] pci 0000:00:18.3: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.373632] pci 0000:00:18.4: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.374299] pci 0000:00:18.4: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.374608] pci 0000:00:18.5: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.375493] pci 0000:00:18.5: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.375802] pci 0000:00:18.6: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.376706] pci 0000:00:18.6: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.377028] pci 0000:00:18.7: [15ad:07a0] type 01 class 0x060400 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.377939] pci 0000:00:18.7: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.378697] pci 0000:00:01.0: PCI bridge to [bus 01] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379367] pci 0000:00:11.0: PCI bridge to [bus 02] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379485] pci 0000:00:11.0: bridge window [io 0x2000-0x3fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379505] pci 0000:00:11.0: bridge window [mem 0xd1900000-0xd23fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379540] pci 0000:00:11.0: bridge window [mem 0xdc400000-0xdc9fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379542] pci 0000:00:11.0: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379544] pci 0000:00:11.0: bridge window [mem 0x000cc000-0x000cffff] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379546] pci 0000:00:11.0: bridge window [mem 0x000d0000-0x000d3fff] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379547] pci 0000:00:11.0: bridge window [mem 0x000d4000-0x000d7fff] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379548] pci 0000:00:11.0: bridge window [mem 0x000d8000-0x000dbfff] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379550] pci 0000:00:11.0: bridge window [mem 0xc0000000-0xfebfffff] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379552] pci 0000:00:11.0: bridge window [io 0x0000-0x0cf7] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.379553] pci 0000:00:11.0: bridge window [io 0x0d00-0xfeff] (subtractive decode) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.380029] pci 0000:03:00.0: [15ad:07c0] type 00 class 0x010700 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.381617] pci 0000:03:00.0: reg 10: [io 0x4008-0x400f] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.383180] pci 0000:03:00.0: reg 14: [mem 0xd2400000-0xd2407fff 64bit] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.389311] pci 0000:03:00.0: reg 30: [mem 0x00000000-0x0000ffff pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.389797] pci 0000:03:00.0: PME# supported from D0 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.389925] pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' Mar 10 09:26:55 lustre-utils01 kernel: [ 0.390120] pci 0000:00:15.0: PCI bridge to [bus 03] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.390219] pci 0000:00:15.0: bridge window [io 0x4000-0x4fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.390238] pci 0000:00:15.0: bridge window [mem 0xd2400000-0xd24fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.390273] pci 0000:00:15.0: bridge window [mem 0xd4400000-0xd44fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.390692] pci 0000:00:15.1: PCI bridge to [bus 04] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.390792] pci 0000:00:15.1: bridge window [io 0x8000-0x8fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.390811] pci 0000:00:15.1: bridge window [mem 0xd2800000-0xd28fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.390846] pci 0000:00:15.1: bridge window [mem 0xd4800000-0xd48fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.391238] pci 0000:00:15.2: PCI bridge to [bus 05] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.391339] pci 0000:00:15.2: bridge window [io 0xc000-0xcfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.391358] pci 0000:00:15.2: bridge window [mem 0xd2c00000-0xd2cfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.391392] pci 0000:00:15.2: bridge window [mem 0xdcb00000-0xdcbfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.391789] pci 0000:00:15.3: PCI bridge to [bus 06] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.391910] pci 0000:00:15.3: bridge window [mem 0xd3000000-0xd30fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.391945] pci 0000:00:15.3: bridge window [mem 0xdcd00000-0xdcdfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.392354] pci 0000:00:15.4: PCI bridge to [bus 07] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.392471] pci 0000:00:15.4: bridge window [mem 0xd3400000-0xd34fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.392506] pci 0000:00:15.4: bridge window [mem 0xdcf00000-0xdcffffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.392904] pci 0000:00:15.5: PCI bridge to [bus 08] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.393020] pci 0000:00:15.5: bridge window [mem 0xd3800000-0xd38fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.393055] pci 0000:00:15.5: bridge window [mem 0xdd100000-0xdd1fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.393454] pci 0000:00:15.6: PCI bridge to [bus 09] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.393594] pci 0000:00:15.6: bridge window [mem 0xd3c00000-0xd3cfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.393634] pci 0000:00:15.6: bridge window [mem 0xdd300000-0xdd3fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.394030] pci 0000:00:15.7: PCI bridge to [bus 0a] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.394164] pci 0000:00:15.7: bridge window [mem 0xd4000000-0xd40fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.394199] pci 0000:00:15.7: bridge window [mem 0xdd500000-0xdd5fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.394679] pci 0000:0b:00.0: [15ad:07b0] type 00 class 0x020000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.395742] pci 0000:0b:00.0: reg 10: [mem 0xd2504000-0xd2504fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.396753] pci 0000:0b:00.0: reg 14: [mem 0xd2503000-0xd2503fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.397831] pci 0000:0b:00.0: reg 18: [mem 0xd2500000-0xd2501fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.399196] pci 0000:0b:00.0: reg 1c: [io 0x5000-0x500f] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.403290] pci 0000:0b:00.0: reg 30: [mem 0x00000000-0x0000ffff pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.403809] pci 0000:0b:00.0: supports D1 D2 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.403813] pci 0000:0b:00.0: PME# supported from D0 D1 D2 D3hot D3cold Mar 10 09:26:55 lustre-utils01 kernel: [ 0.404022] pci 0000:0b:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' Mar 10 09:26:55 lustre-utils01 kernel: [ 0.404306] pci 0000:00:16.0: PCI bridge to [bus 0b] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.404454] pci 0000:00:16.0: bridge window [io 0x5000-0x5fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.404485] pci 0000:00:16.0: bridge window [mem 0xd2500000-0xd25fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.404540] pci 0000:00:16.0: bridge window [mem 0xd4500000-0xd45fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.405173] pci 0000:00:16.1: PCI bridge to [bus 0c] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.405320] pci 0000:00:16.1: bridge window [io 0x9000-0x9fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.405350] pci 0000:00:16.1: bridge window [mem 0xd2900000-0xd29fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.405406] pci 0000:00:16.1: bridge window [mem 0xd4900000-0xd49fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.406032] pci 0000:00:16.2: PCI bridge to [bus 0d] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.406175] pci 0000:00:16.2: bridge window [io 0xd000-0xdfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.406205] pci 0000:00:16.2: bridge window [mem 0xd2d00000-0xd2dfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.406257] pci 0000:00:16.2: bridge window [mem 0xd4b00000-0xd4bfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.406857] pci 0000:00:16.3: PCI bridge to [bus 0e] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.407031] pci 0000:00:16.3: bridge window [mem 0xd3100000-0xd31fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.407086] pci 0000:00:16.3: bridge window [mem 0xd4d00000-0xd4dfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.407652] pci 0000:00:16.4: PCI bridge to [bus 0f] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.407828] pci 0000:00:16.4: bridge window [mem 0xd3500000-0xd35fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.407882] pci 0000:00:16.4: bridge window [mem 0xd4f00000-0xd4ffffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.408478] pci 0000:00:16.5: PCI bridge to [bus 10] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.408651] pci 0000:00:16.5: bridge window [mem 0xd3900000-0xd39fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.408709] pci 0000:00:16.5: bridge window [mem 0xd5100000-0xd51fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.409307] pci 0000:00:16.6: PCI bridge to [bus 11] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.409498] pci 0000:00:16.6: bridge window [mem 0xd3d00000-0xd3dfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.409555] pci 0000:00:16.6: bridge window [mem 0xd5300000-0xd53fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.410148] pci 0000:00:16.7: PCI bridge to [bus 12] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.410312] pci 0000:00:16.7: bridge window [mem 0xd4100000-0xd41fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.410368] pci 0000:00:16.7: bridge window [mem 0xd5500000-0xd55fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.410970] pci 0000:00:17.0: PCI bridge to [bus 13] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.411113] pci 0000:00:17.0: bridge window [io 0x6000-0x6fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.411143] pci 0000:00:17.0: bridge window [mem 0xd2600000-0xd26fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.411192] pci 0000:00:17.0: bridge window [mem 0xd4600000-0xd46fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.411789] pci 0000:00:17.1: PCI bridge to [bus 14] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.411936] pci 0000:00:17.1: bridge window [io 0xa000-0xafff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.411967] pci 0000:00:17.1: bridge window [mem 0xd2a00000-0xd2afffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.412023] pci 0000:00:17.1: bridge window [mem 0xdca00000-0xdcafffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.412455] pci 0000:00:17.2: PCI bridge to [bus 15] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.412556] pci 0000:00:17.2: bridge window [io 0xe000-0xefff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.412575] pci 0000:00:17.2: bridge window [mem 0xd2e00000-0xd2efffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.412610] pci 0000:00:17.2: bridge window [mem 0xdcc00000-0xdccfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.412993] pci 0000:00:17.3: PCI bridge to [bus 16] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.413132] pci 0000:00:17.3: bridge window [mem 0xd3200000-0xd32fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.413180] pci 0000:00:17.3: bridge window [mem 0xdce00000-0xdcefffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.413585] pci 0000:00:17.4: PCI bridge to [bus 17] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.413702] pci 0000:00:17.4: bridge window [mem 0xd3600000-0xd36fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.413738] pci 0000:00:17.4: bridge window [mem 0xdd000000-0xdd0fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.414153] pci 0000:00:17.5: PCI bridge to [bus 18] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.414270] pci 0000:00:17.5: bridge window [mem 0xd3a00000-0xd3afffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.414305] pci 0000:00:17.5: bridge window [mem 0xdd200000-0xdd2fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.414688] pci 0000:00:17.6: PCI bridge to [bus 19] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.414806] pci 0000:00:17.6: bridge window [mem 0xd3e00000-0xd3efffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.414843] pci 0000:00:17.6: bridge window [mem 0xdd400000-0xdd4fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.415332] pci 0000:00:17.7: PCI bridge to [bus 1a] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.415532] pci 0000:00:17.7: bridge window [mem 0xd4200000-0xd42fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.415588] pci 0000:00:17.7: bridge window [mem 0xdd600000-0xdd6fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.416185] pci 0000:00:18.0: PCI bridge to [bus 1b] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.417981] pci 0000:00:18.0: bridge window [io 0x7000-0x7fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.418011] pci 0000:00:18.0: bridge window [mem 0xd2700000-0xd27fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.418060] pci 0000:00:18.0: bridge window [mem 0xd4700000-0xd47fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.418496] pci 0000:00:18.1: PCI bridge to [bus 1c] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.418598] pci 0000:00:18.1: bridge window [io 0xb000-0xbfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.418617] pci 0000:00:18.1: bridge window [mem 0xd2b00000-0xd2bfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.418652] pci 0000:00:18.1: bridge window [mem 0xd4a00000-0xd4afffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.419072] pci 0000:00:18.2: PCI bridge to [bus 1d] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.419193] pci 0000:00:18.2: bridge window [io 0xf000-0xffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.419212] pci 0000:00:18.2: bridge window [mem 0xd2f00000-0xd2ffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.419247] pci 0000:00:18.2: bridge window [mem 0xd4c00000-0xd4cfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.419633] pci 0000:00:18.3: PCI bridge to [bus 1e] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.419749] pci 0000:00:18.3: bridge window [mem 0xd3300000-0xd33fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.419793] pci 0000:00:18.3: bridge window [mem 0xd4e00000-0xd4efffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.420207] pci 0000:00:18.4: PCI bridge to [bus 1f] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.420325] pci 0000:00:18.4: bridge window [mem 0xd3700000-0xd37fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.420360] pci 0000:00:18.4: bridge window [mem 0xd5000000-0xd50fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.420749] pci 0000:00:18.5: PCI bridge to [bus 20] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.420866] pci 0000:00:18.5: bridge window [mem 0xd3b00000-0xd3bfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.420901] pci 0000:00:18.5: bridge window [mem 0xd5200000-0xd52fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.421495] pci 0000:00:18.6: PCI bridge to [bus 21] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.421671] pci 0000:00:18.6: bridge window [mem 0xd3f00000-0xd3ffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.421726] pci 0000:00:18.6: bridge window [mem 0xd5400000-0xd54fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.422312] pci 0000:00:18.7: PCI bridge to [bus 22] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.422486] pci 0000:00:18.7: bridge window [mem 0xd4300000-0xd43fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.422541] pci 0000:00:18.7: bridge window [mem 0xd5600000-0xd56fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.424723] pci0000:00: Requesting ACPI _OSC control (0x1d) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.425128] pci0000:00: ACPI _OSC control (0x15) granted Mar 10 09:26:55 lustre-utils01 kernel: [ 0.427796] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *9 10 11 14 15) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.428258] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 14 15) *0, disabled. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.428773] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 14 15) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.429234] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 14 15) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.429768] vgaarb: device added: PCI:0000:00:0f.0,decodes=io+mem,owns=io+mem,locks=none Mar 10 09:26:55 lustre-utils01 kernel: [ 0.432157] vgaarb: loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.432219] vgaarb: bridge control possible 0000:00:0f.0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.432480] SCSI subsystem initialized Mar 10 09:26:55 lustre-utils01 kernel: [ 0.432554] ACPI: bus type scsi registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.432719] libata version 3.00 loaded. Mar 10 09:26:55 lustre-utils01 kernel: [ 0.432737] ACPI: bus type usb registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.432831] usbcore: registered new interface driver usbfs Mar 10 09:26:55 lustre-utils01 kernel: [ 0.432930] usbcore: registered new interface driver hub Mar 10 09:26:55 lustre-utils01 kernel: [ 0.433055] usbcore: registered new device driver usb Mar 10 09:26:55 lustre-utils01 kernel: [ 0.433222] PCI: Using ACPI for IRQ routing Mar 10 09:26:55 lustre-utils01 kernel: [ 0.508355] PCI: pci_cache_line_size set to 64 bytes Mar 10 09:26:55 lustre-utils01 kernel: [ 0.508390] pci 0000:00:18.2: no compatible bridge window for [io 0xf000-0xffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.509745] e820: reserve RAM buffer [mem 0x0009f800-0x0009ffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.509749] e820: reserve RAM buffer [mem 0xbfef0000-0xbfffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.509863] NetLabel: Initializing Mar 10 09:26:55 lustre-utils01 kernel: [ 0.509935] NetLabel: domain hash size = 128 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.510009] NetLabel: protocols = UNLABELED CIPSOv4 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.510100] NetLabel: unlabeled traffic allowed by default Mar 10 09:26:55 lustre-utils01 kernel: [ 0.510352] HPET: 16 timers in total, 0 timers will be used for per-cpu timer Mar 10 09:26:55 lustre-utils01 kernel: [ 0.510511] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.511039] hpet0: 16 comparators, 64-bit 14.318180 MHz counter Mar 10 09:26:55 lustre-utils01 kernel: [ 0.513242] Switching to clocksource hpet Mar 10 09:26:55 lustre-utils01 kernel: [ 0.520466] AppArmor: AppArmor Filesystem Enabled Mar 10 09:26:55 lustre-utils01 kernel: [ 0.520592] pnp: PnP ACPI init Mar 10 09:26:55 lustre-utils01 kernel: [ 0.520668] ACPI: bus type pnp registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.520932] system 00:00: [io 0x1000-0x103f] has been reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521025] system 00:00: [io 0x1040-0x104f] has been reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521125] system 00:00: [io 0x0cf0-0x0cf1] has been reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521217] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521234] pnp 00:01: [dma 4] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521249] pnp 00:01: Plug and Play ACPI device, IDs PNP0200 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521269] pnp 00:02: Plug and Play ACPI device, IDs PNP0001 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521800] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521817] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521843] pnp 00:05: Plug and Play ACPI device, IDs PNP0303 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.521865] pnp 00:06: Plug and Play ACPI device, IDs PNP0f13 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.522083] system 00:07: [mem 0xfed00000-0xfed003ff] has been reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.522186] system 00:07: Plug and Play ACPI device, IDs PNP0103 PNP0c01 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.530517] pnp 00:08: Plug and Play ACPI device, IDs PNP0400 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.539321] pnp 00:09: Plug and Play ACPI device, IDs PNP0501 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.545249] pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.552407] pnp 00:0b: [dma 2] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.554069] pnp 00:0b: Plug and Play ACPI device, IDs PNP0700 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.554410] system 00:0c: [io 0x1060-0x107f] has been reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.554557] system 00:0c: [mem 0xe0000000-0xefffffff] has been reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.554690] system 00:0c: [mem 0xd0200000-0xd03fffff] has been reserved Mar 10 09:26:55 lustre-utils01 kernel: [ 0.554829] system 00:0c: Plug and Play ACPI device, IDs PNP0c02 (active) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.561942] pnp: PnP ACPI: found 13 devices Mar 10 09:26:55 lustre-utils01 kernel: [ 0.562062] ACPI: ACPI bus type pnp unregistered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.568814] pci 0000:00:15.3: bridge window [io 0x1000-0x0fff] to [bus 06] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.568900] pci 0000:00:15.4: bridge window [io 0x1000-0x0fff] to [bus 07] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.568983] pci 0000:00:15.5: bridge window [io 0x1000-0x0fff] to [bus 08] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.569066] pci 0000:00:15.6: bridge window [io 0x1000-0x0fff] to [bus 09] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.569148] pci 0000:00:15.7: bridge window [io 0x1000-0x0fff] to [bus 0a] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.569499] pci 0000:00:16.3: bridge window [io 0x1000-0x0fff] to [bus 0e] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.569581] pci 0000:00:16.4: bridge window [io 0x1000-0x0fff] to [bus 0f] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.569665] pci 0000:00:16.5: bridge window [io 0x1000-0x0fff] to [bus 10] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.569749] pci 0000:00:16.6: bridge window [io 0x1000-0x0fff] to [bus 11] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.569822] pci 0000:00:16.7: bridge window [io 0x1000-0x0fff] to [bus 12] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570119] pci 0000:00:17.3: bridge window [io 0x1000-0x0fff] to [bus 16] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570204] pci 0000:00:17.4: bridge window [io 0x1000-0x0fff] to [bus 17] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570284] pci 0000:00:17.5: bridge window [io 0x1000-0x0fff] to [bus 18] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570358] pci 0000:00:17.6: bridge window [io 0x1000-0x0fff] to [bus 19] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570442] pci 0000:00:17.7: bridge window [io 0x1000-0x0fff] to [bus 1a] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570680] pci 0000:00:18.2: bridge window [io 0x1000-0x0fff] to [bus 1d] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570761] pci 0000:00:18.3: bridge window [io 0x1000-0x0fff] to [bus 1e] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570843] pci 0000:00:18.4: bridge window [io 0x1000-0x0fff] to [bus 1f] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.570925] pci 0000:00:18.5: bridge window [io 0x1000-0x0fff] to [bus 20] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571007] pci 0000:00:18.6: bridge window [io 0x1000-0x0fff] to [bus 21] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571079] pci 0000:00:18.7: bridge window [io 0x1000-0x0fff] to [bus 22] add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571106] pci 0000:00:15.3: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571109] pci 0000:00:15.4: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571111] pci 0000:00:15.5: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571114] pci 0000:00:15.6: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571116] pci 0000:00:15.7: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571119] pci 0000:00:16.3: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571121] pci 0000:00:16.4: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571123] pci 0000:00:16.5: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571125] pci 0000:00:16.6: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571128] pci 0000:00:16.7: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571130] pci 0000:00:17.3: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571133] pci 0000:00:17.4: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571135] pci 0000:00:17.5: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571138] pci 0000:00:17.6: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571141] pci 0000:00:17.7: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571143] pci 0000:00:18.2: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571145] pci 0000:00:18.3: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571148] pci 0000:00:18.4: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571150] pci 0000:00:18.5: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571153] pci 0000:00:18.6: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571155] pci 0000:00:18.7: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571162] pci 0000:00:0f.0: BAR 6: assigned [mem 0xc0000000-0xc0007fff pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571319] pci 0000:00:15.3: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571413] pci 0000:00:15.4: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571506] pci 0000:00:15.5: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571597] pci 0000:00:15.6: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571690] pci 0000:00:15.7: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571784] pci 0000:00:16.3: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.571875] pci 0000:00:16.4: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572004] pci 0000:00:16.5: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572115] pci 0000:00:16.6: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572209] pci 0000:00:16.7: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572301] pci 0000:00:17.3: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572394] pci 0000:00:17.4: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572486] pci 0000:00:17.5: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572578] pci 0000:00:17.6: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572686] pci 0000:00:17.7: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572823] pci 0000:00:18.2: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.572927] pci 0000:00:18.3: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.573056] pci 0000:00:18.4: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.573148] pci 0000:00:18.5: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.573241] pci 0000:00:18.6: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.573363] pci 0000:00:18.7: BAR 13: can't assign io (size 0x1000) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.573460] pci 0000:00:01.0: PCI bridge to [bus 01] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.573623] pci 0000:00:11.0: PCI bridge to [bus 02] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.573714] pci 0000:00:11.0: bridge window [io 0x2000-0x3fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.575955] pci 0000:00:11.0: bridge window [mem 0xd1900000-0xd23fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.576099] pci 0000:00:11.0: bridge window [mem 0xdc400000-0xdc9fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.576279] pci 0000:03:00.0: BAR 6: assigned [mem 0xd4400000-0xd440ffff pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.576396] pci 0000:00:15.0: PCI bridge to [bus 03] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.576500] pci 0000:00:15.0: bridge window [io 0x4000-0x4fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.576650] pci 0000:00:15.0: bridge window [mem 0xd2400000-0xd24fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.576764] pci 0000:00:15.0: bridge window [mem 0xd4400000-0xd44fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.576964] pci 0000:00:15.1: PCI bridge to [bus 04] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.577096] pci 0000:00:15.1: bridge window [io 0x8000-0x8fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.577227] pci 0000:00:15.1: bridge window [mem 0xd2800000-0xd28fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.577373] pci 0000:00:15.1: bridge window [mem 0xd4800000-0xd48fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.577546] pci 0000:00:15.2: PCI bridge to [bus 05] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.577638] pci 0000:00:15.2: bridge window [io 0xc000-0xcfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.577755] pci 0000:00:15.2: bridge window [mem 0xd2c00000-0xd2cfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.577870] pci 0000:00:15.2: bridge window [mem 0xdcb00000-0xdcbfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.578044] pci 0000:00:15.3: PCI bridge to [bus 06] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.578171] pci 0000:00:15.3: bridge window [mem 0xd3000000-0xd30fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.578286] pci 0000:00:15.3: bridge window [mem 0xdcd00000-0xdcdfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.578455] pci 0000:00:15.4: PCI bridge to [bus 07] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.578563] pci 0000:00:15.4: bridge window [mem 0xd3400000-0xd34fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.578691] pci 0000:00:15.4: bridge window [mem 0xdcf00000-0xdcffffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.578878] pci 0000:00:15.5: PCI bridge to [bus 08] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.579008] pci 0000:00:15.5: bridge window [mem 0xd3800000-0xd38fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.579165] pci 0000:00:15.5: bridge window [mem 0xdd100000-0xdd1fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.579336] pci 0000:00:15.6: PCI bridge to [bus 09] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.579445] pci 0000:00:15.6: bridge window [mem 0xd3c00000-0xd3cfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.579560] pci 0000:00:15.6: bridge window [mem 0xdd300000-0xdd3fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.579725] pci 0000:00:15.7: PCI bridge to [bus 0a] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.579833] pci 0000:00:15.7: bridge window [mem 0xd4000000-0xd40fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.579947] pci 0000:00:15.7: bridge window [mem 0xdd500000-0xdd5fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.580217] pci 0000:0b:00.0: BAR 6: assigned [mem 0xd4500000-0xd450ffff pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.580335] pci 0000:00:16.0: PCI bridge to [bus 0b] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.580427] pci 0000:00:16.0: bridge window [io 0x5000-0x5fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.580543] pci 0000:00:16.0: bridge window [mem 0xd2500000-0xd25fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.580695] pci 0000:00:16.0: bridge window [mem 0xd4500000-0xd45fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.580895] pci 0000:00:16.1: PCI bridge to [bus 0c] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.580994] pci 0000:00:16.1: bridge window [io 0x9000-0x9fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.581130] pci 0000:00:16.1: bridge window [mem 0xd2900000-0xd29fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.581245] pci 0000:00:16.1: bridge window [mem 0xd4900000-0xd49fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.581692] pci 0000:00:16.2: PCI bridge to [bus 0d] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.581808] pci 0000:00:16.2: bridge window [io 0xd000-0xdfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.581980] pci 0000:00:16.2: bridge window [mem 0xd2d00000-0xd2dfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.582146] pci 0000:00:16.2: bridge window [mem 0xd4b00000-0xd4bfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.582395] pci 0000:00:16.3: PCI bridge to [bus 0e] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.582558] pci 0000:00:16.3: bridge window [mem 0xd3100000-0xd31fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.582722] pci 0000:00:16.3: bridge window [mem 0xd4d00000-0xd4dfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.582921] pci 0000:00:16.4: PCI bridge to [bus 0f] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.583067] pci 0000:00:16.4: bridge window [mem 0xd3500000-0xd35fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.583229] pci 0000:00:16.4: bridge window [mem 0xd4f00000-0xd4ffffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.583478] pci 0000:00:16.5: PCI bridge to [bus 10] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.583633] pci 0000:00:16.5: bridge window [mem 0xd3900000-0xd39fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.583801] pci 0000:00:16.5: bridge window [mem 0xd5100000-0xd51fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.584048] pci 0000:00:16.6: PCI bridge to [bus 11] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.584207] pci 0000:00:16.6: bridge window [mem 0xd3d00000-0xd3dfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.584377] pci 0000:00:16.6: bridge window [mem 0xd5300000-0xd53fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.584625] pci 0000:00:16.7: PCI bridge to [bus 12] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.584785] pci 0000:00:16.7: bridge window [mem 0xd4100000-0xd41fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.584946] pci 0000:00:16.7: bridge window [mem 0xd5500000-0xd55fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.585189] pci 0000:00:17.0: PCI bridge to [bus 13] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.585356] pci 0000:00:17.0: bridge window [io 0x6000-0x6fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.585530] pci 0000:00:17.0: bridge window [mem 0xd2600000-0xd26fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.585700] pci 0000:00:17.0: bridge window [mem 0xd4600000-0xd46fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.585953] pci 0000:00:17.1: PCI bridge to [bus 14] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.586084] pci 0000:00:17.1: bridge window [io 0xa000-0xafff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.586260] pci 0000:00:17.1: bridge window [mem 0xd2a00000-0xd2afffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.586431] pci 0000:00:17.1: bridge window [mem 0xdca00000-0xdcafffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.586681] pci 0000:00:17.2: PCI bridge to [bus 15] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.586815] pci 0000:00:17.2: bridge window [io 0xe000-0xefff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.586985] pci 0000:00:17.2: bridge window [mem 0xd2e00000-0xd2efffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.587147] pci 0000:00:17.2: bridge window [mem 0xdcc00000-0xdccfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.587392] pci 0000:00:17.3: PCI bridge to [bus 16] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.587555] pci 0000:00:17.3: bridge window [mem 0xd3200000-0xd32fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.587726] pci 0000:00:17.3: bridge window [mem 0xdce00000-0xdcefffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.587970] pci 0000:00:17.4: PCI bridge to [bus 17] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.588111] pci 0000:00:17.4: bridge window [mem 0xd3600000-0xd36fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.590433] pci 0000:00:17.4: bridge window [mem 0xdd000000-0xdd0fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.590621] pci 0000:00:17.5: PCI bridge to [bus 18] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.590736] pci 0000:00:17.5: bridge window [mem 0xd3a00000-0xd3afffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.590892] pci 0000:00:17.5: bridge window [mem 0xdd200000-0xdd2fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.591099] pci 0000:00:17.6: PCI bridge to [bus 19] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.591209] pci 0000:00:17.6: bridge window [mem 0xd3e00000-0xd3efffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.591323] pci 0000:00:17.6: bridge window [mem 0xdd400000-0xdd4fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.591489] pci 0000:00:17.7: PCI bridge to [bus 1a] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.591597] pci 0000:00:17.7: bridge window [mem 0xd4200000-0xd42fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.591717] pci 0000:00:17.7: bridge window [mem 0xdd600000-0xdd6fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.591898] pci 0000:00:18.0: PCI bridge to [bus 1b] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.592001] pci 0000:00:18.0: bridge window [io 0x7000-0x7fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.592134] pci 0000:00:18.0: bridge window [mem 0xd2700000-0xd27fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.592250] pci 0000:00:18.0: bridge window [mem 0xd4700000-0xd47fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.592419] pci 0000:00:18.1: PCI bridge to [bus 1c] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.592512] pci 0000:00:18.1: bridge window [io 0xb000-0xbfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.592628] pci 0000:00:18.1: bridge window [mem 0xd2b00000-0xd2bfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.592760] pci 0000:00:18.1: bridge window [mem 0xd4a00000-0xd4afffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.592928] pci 0000:00:18.2: PCI bridge to [bus 1d] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.593059] pci 0000:00:18.2: bridge window [mem 0xd2f00000-0xd2ffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.593266] pci 0000:00:18.2: bridge window [mem 0xd4c00000-0xd4cfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.593453] pci 0000:00:18.3: PCI bridge to [bus 1e] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.593557] pci 0000:00:18.3: bridge window [mem 0xd3300000-0xd33fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.593666] pci 0000:00:18.3: bridge window [mem 0xd4e00000-0xd4efffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.593851] pci 0000:00:18.4: PCI bridge to [bus 1f] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.593961] pci 0000:00:18.4: bridge window [mem 0xd3700000-0xd37fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.594099] pci 0000:00:18.4: bridge window [mem 0xd5000000-0xd50fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.594270] pci 0000:00:18.5: PCI bridge to [bus 20] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.594374] pci 0000:00:18.5: bridge window [mem 0xd3b00000-0xd3bfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.594483] pci 0000:00:18.5: bridge window [mem 0xd5200000-0xd52fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.594669] pci 0000:00:18.6: PCI bridge to [bus 21] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.594775] pci 0000:00:18.6: bridge window [mem 0xd3f00000-0xd3ffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.594884] pci 0000:00:18.6: bridge window [mem 0xd5400000-0xd54fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.595066] pci 0000:00:18.7: PCI bridge to [bus 22] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.595170] pci 0000:00:18.7: bridge window [mem 0xd4300000-0xd43fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.595279] pci 0000:00:18.7: bridge window [mem 0xd5600000-0xd56fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.595480] pci 0000:00:01.0: setting latency timer to 64 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596331] pci_bus 0000:00: resource 4 [mem 0x000a0000-0x000bffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596333] pci_bus 0000:00: resource 5 [mem 0x000cc000-0x000cffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596334] pci_bus 0000:00: resource 6 [mem 0x000d0000-0x000d3fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596336] pci_bus 0000:00: resource 7 [mem 0x000d4000-0x000d7fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596337] pci_bus 0000:00: resource 8 [mem 0x000d8000-0x000dbfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596339] pci_bus 0000:00: resource 9 [mem 0xc0000000-0xfebfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596340] pci_bus 0000:00: resource 10 [io 0x0000-0x0cf7] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596342] pci_bus 0000:00: resource 11 [io 0x0d00-0xfeff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596344] pci_bus 0000:02: resource 0 [io 0x2000-0x3fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596345] pci_bus 0000:02: resource 1 [mem 0xd1900000-0xd23fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596347] pci_bus 0000:02: resource 2 [mem 0xdc400000-0xdc9fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596348] pci_bus 0000:02: resource 4 [mem 0x000a0000-0x000bffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596350] pci_bus 0000:02: resource 5 [mem 0x000cc000-0x000cffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596351] pci_bus 0000:02: resource 6 [mem 0x000d0000-0x000d3fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596353] pci_bus 0000:02: resource 7 [mem 0x000d4000-0x000d7fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596354] pci_bus 0000:02: resource 8 [mem 0x000d8000-0x000dbfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596355] pci_bus 0000:02: resource 9 [mem 0xc0000000-0xfebfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596357] pci_bus 0000:02: resource 10 [io 0x0000-0x0cf7] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596358] pci_bus 0000:02: resource 11 [io 0x0d00-0xfeff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596360] pci_bus 0000:03: resource 0 [io 0x4000-0x4fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596361] pci_bus 0000:03: resource 1 [mem 0xd2400000-0xd24fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596363] pci_bus 0000:03: resource 2 [mem 0xd4400000-0xd44fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596364] pci_bus 0000:04: resource 0 [io 0x8000-0x8fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596366] pci_bus 0000:04: resource 1 [mem 0xd2800000-0xd28fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596367] pci_bus 0000:04: resource 2 [mem 0xd4800000-0xd48fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596369] pci_bus 0000:05: resource 0 [io 0xc000-0xcfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596370] pci_bus 0000:05: resource 1 [mem 0xd2c00000-0xd2cfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596372] pci_bus 0000:05: resource 2 [mem 0xdcb00000-0xdcbfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596373] pci_bus 0000:06: resource 1 [mem 0xd3000000-0xd30fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596375] pci_bus 0000:06: resource 2 [mem 0xdcd00000-0xdcdfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596377] pci_bus 0000:07: resource 1 [mem 0xd3400000-0xd34fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596378] pci_bus 0000:07: resource 2 [mem 0xdcf00000-0xdcffffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596380] pci_bus 0000:08: resource 1 [mem 0xd3800000-0xd38fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596381] pci_bus 0000:08: resource 2 [mem 0xdd100000-0xdd1fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596383] pci_bus 0000:09: resource 1 [mem 0xd3c00000-0xd3cfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596384] pci_bus 0000:09: resource 2 [mem 0xdd300000-0xdd3fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596386] pci_bus 0000:0a: resource 1 [mem 0xd4000000-0xd40fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596387] pci_bus 0000:0a: resource 2 [mem 0xdd500000-0xdd5fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596389] pci_bus 0000:0b: resource 0 [io 0x5000-0x5fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596390] pci_bus 0000:0b: resource 1 [mem 0xd2500000-0xd25fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596392] pci_bus 0000:0b: resource 2 [mem 0xd4500000-0xd45fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596393] pci_bus 0000:0c: resource 0 [io 0x9000-0x9fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596394] pci_bus 0000:0c: resource 1 [mem 0xd2900000-0xd29fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596396] pci_bus 0000:0c: resource 2 [mem 0xd4900000-0xd49fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596397] pci_bus 0000:0d: resource 0 [io 0xd000-0xdfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596399] pci_bus 0000:0d: resource 1 [mem 0xd2d00000-0xd2dfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596400] pci_bus 0000:0d: resource 2 [mem 0xd4b00000-0xd4bfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596402] pci_bus 0000:0e: resource 1 [mem 0xd3100000-0xd31fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596403] pci_bus 0000:0e: resource 2 [mem 0xd4d00000-0xd4dfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596405] pci_bus 0000:0f: resource 1 [mem 0xd3500000-0xd35fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596406] pci_bus 0000:0f: resource 2 [mem 0xd4f00000-0xd4ffffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596408] pci_bus 0000:10: resource 1 [mem 0xd3900000-0xd39fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596410] pci_bus 0000:10: resource 2 [mem 0xd5100000-0xd51fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596411] pci_bus 0000:11: resource 1 [mem 0xd3d00000-0xd3dfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596413] pci_bus 0000:11: resource 2 [mem 0xd5300000-0xd53fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596414] pci_bus 0000:12: resource 1 [mem 0xd4100000-0xd41fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596416] pci_bus 0000:12: resource 2 [mem 0xd5500000-0xd55fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596417] pci_bus 0000:13: resource 0 [io 0x6000-0x6fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596419] pci_bus 0000:13: resource 1 [mem 0xd2600000-0xd26fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596420] pci_bus 0000:13: resource 2 [mem 0xd4600000-0xd46fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596422] pci_bus 0000:14: resource 0 [io 0xa000-0xafff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596423] pci_bus 0000:14: resource 1 [mem 0xd2a00000-0xd2afffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596425] pci_bus 0000:14: resource 2 [mem 0xdca00000-0xdcafffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596426] pci_bus 0000:15: resource 0 [io 0xe000-0xefff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596428] pci_bus 0000:15: resource 1 [mem 0xd2e00000-0xd2efffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596429] pci_bus 0000:15: resource 2 [mem 0xdcc00000-0xdccfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596431] pci_bus 0000:16: resource 1 [mem 0xd3200000-0xd32fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596432] pci_bus 0000:16: resource 2 [mem 0xdce00000-0xdcefffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596434] pci_bus 0000:17: resource 1 [mem 0xd3600000-0xd36fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596435] pci_bus 0000:17: resource 2 [mem 0xdd000000-0xdd0fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596437] pci_bus 0000:18: resource 1 [mem 0xd3a00000-0xd3afffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596439] pci_bus 0000:18: resource 2 [mem 0xdd200000-0xdd2fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596440] pci_bus 0000:19: resource 1 [mem 0xd3e00000-0xd3efffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596442] pci_bus 0000:19: resource 2 [mem 0xdd400000-0xdd4fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596443] pci_bus 0000:1a: resource 1 [mem 0xd4200000-0xd42fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596445] pci_bus 0000:1a: resource 2 [mem 0xdd600000-0xdd6fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596446] pci_bus 0000:1b: resource 0 [io 0x7000-0x7fff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596448] pci_bus 0000:1b: resource 1 [mem 0xd2700000-0xd27fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596449] pci_bus 0000:1b: resource 2 [mem 0xd4700000-0xd47fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596451] pci_bus 0000:1c: resource 0 [io 0xb000-0xbfff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596452] pci_bus 0000:1c: resource 1 [mem 0xd2b00000-0xd2bfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596453] pci_bus 0000:1c: resource 2 [mem 0xd4a00000-0xd4afffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596455] pci_bus 0000:1d: resource 1 [mem 0xd2f00000-0xd2ffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596456] pci_bus 0000:1d: resource 2 [mem 0xd4c00000-0xd4cfffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596458] pci_bus 0000:1e: resource 1 [mem 0xd3300000-0xd33fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596459] pci_bus 0000:1e: resource 2 [mem 0xd4e00000-0xd4efffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596461] pci_bus 0000:1f: resource 1 [mem 0xd3700000-0xd37fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596462] pci_bus 0000:1f: resource 2 [mem 0xd5000000-0xd50fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596464] pci_bus 0000:20: resource 1 [mem 0xd3b00000-0xd3bfffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596466] pci_bus 0000:20: resource 2 [mem 0xd5200000-0xd52fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596467] pci_bus 0000:21: resource 1 [mem 0xd3f00000-0xd3ffffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596469] pci_bus 0000:21: resource 2 [mem 0xd5400000-0xd54fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596470] pci_bus 0000:22: resource 1 [mem 0xd4300000-0xd43fffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596472] pci_bus 0000:22: resource 2 [mem 0xd5600000-0xd56fffff 64bit pref] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596518] NET: Registered protocol family 2 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.596837] TCP established hash table entries: 131072 (order: 9, 2097152 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.597623] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.599104] TCP: Hash tables configured (established 131072 bind 65536) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.599227] TCP: reno registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.599314] UDP hash table entries: 8192 (order: 6, 262144 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.599458] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.600166] NET: Registered protocol family 1 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.600261] pci 0000:00:00.0: Limiting direct PCI/PCI transfers Mar 10 09:26:55 lustre-utils01 kernel: [ 0.600429] pci 0000:00:0f.0: Boot video device Mar 10 09:26:55 lustre-utils01 kernel: [ 0.600749] PCI: CLS 32 bytes, default 64 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.600793] Trying to unpack rootfs image as initramfs... Mar 10 09:26:55 lustre-utils01 kernel: [ 0.901760] Freeing initrd memory: 16376k freed Mar 10 09:26:55 lustre-utils01 kernel: [ 0.905078] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.906635] software IO TLB [mem 0xbbef0000-0xbfef0000] (64MB) mapped at [ffff8800bbef0000-ffff8800bfeeffff] Mar 10 09:26:55 lustre-utils01 kernel: [ 0.906856] Simple Boot Flag at 0x36 set to 0x80 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.906947] Switching to clocksource tsc Mar 10 09:26:55 lustre-utils01 kernel: [ 0.907324] Initialise module verification Mar 10 09:26:55 lustre-utils01 kernel: [ 0.907465] audit: initializing netlink socket (disabled) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.907579] type=2000 audit(1394443604.773:1): initialized Mar 10 09:26:55 lustre-utils01 kernel: [ 0.938389] HugeTLB registered 2 MB page size, pre-allocated 0 pages Mar 10 09:26:55 lustre-utils01 kernel: [ 0.940073] VFS: Disk quotas dquot_6.5.2 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.940181] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.940674] fuse init (API version 7.20) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.940803] msgmni has been set to 32094 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.941104] Key type asymmetric registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.941225] Asymmetric key parser 'x509' registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.941331] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.941466] io scheduler noop registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.941536] io scheduler deadline registered (default) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.941633] io scheduler cfq registered Mar 10 09:26:55 lustre-utils01 kernel: [ 0.942327] pcieport 0000:00:15.0: irq 40 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.943108] pcieport 0000:00:15.1: irq 41 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.943896] pcieport 0000:00:15.2: irq 42 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.944672] pcieport 0000:00:15.3: irq 43 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.945476] pcieport 0000:00:15.4: irq 44 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.946298] pcieport 0000:00:15.5: irq 45 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.947063] pcieport 0000:00:15.6: irq 46 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.947839] pcieport 0000:00:15.7: irq 47 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.948669] pcieport 0000:00:16.0: irq 48 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.949501] pcieport 0000:00:16.1: irq 49 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.950298] pcieport 0000:00:16.2: irq 50 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.951106] pcieport 0000:00:16.3: irq 51 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.951884] pcieport 0000:00:16.4: irq 52 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.952686] pcieport 0000:00:16.5: irq 53 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.953511] pcieport 0000:00:16.6: irq 54 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.954318] pcieport 0000:00:16.7: irq 55 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.955211] pcieport 0000:00:17.0: irq 56 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.956322] pcieport 0000:00:17.1: irq 57 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.957509] pcieport 0000:00:17.2: irq 58 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.958531] pcieport 0000:00:17.3: irq 59 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.959314] pcieport 0000:00:17.4: irq 60 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.960103] pcieport 0000:00:17.5: irq 61 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.960872] pcieport 0000:00:17.6: irq 62 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.961664] pcieport 0000:00:17.7: irq 63 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.962433] pcieport 0000:00:18.0: irq 64 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.963202] pcieport 0000:00:18.1: irq 65 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.963980] pcieport 0000:00:18.2: irq 66 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.964733] pcieport 0000:00:18.3: irq 67 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.965504] pcieport 0000:00:18.4: irq 68 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.966289] pcieport 0000:00:18.5: irq 69 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.967059] pcieport 0000:00:18.6: irq 70 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.967825] pcieport 0000:00:18.7: irq 71 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 0.968257] pcieport 0000:00:15.0: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.968361] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.968469] pcie_pme 0000:00:15.0:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.968591] pcieport 0000:00:15.1: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.968706] pcie_pme 0000:00:15.1:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.968797] pcieport 0000:00:15.2: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.968923] pcie_pme 0000:00:15.2:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969021] pcieport 0000:00:15.3: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969154] pcie_pme 0000:00:15.3:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969246] pcieport 0000:00:15.4: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969357] pcie_pme 0000:00:15.4:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969448] pcieport 0000:00:15.5: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969586] pcie_pme 0000:00:15.5:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969683] pcieport 0000:00:15.6: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969797] pcie_pme 0000:00:15.6:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.969909] pcieport 0000:00:15.7: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970030] pcie_pme 0000:00:15.7:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970123] pcieport 0000:00:16.0: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970222] pci 0000:0b:00.0: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970338] pcie_pme 0000:00:16.0:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970434] pcieport 0000:00:16.1: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970564] pcie_pme 0000:00:16.1:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970657] pcieport 0000:00:16.2: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970778] pcie_pme 0000:00:16.2:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970877] pcieport 0000:00:16.3: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.970995] pcie_pme 0000:00:16.3:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971090] pcieport 0000:00:16.4: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971213] pcie_pme 0000:00:16.4:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971306] pcieport 0000:00:16.5: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971419] pcie_pme 0000:00:16.5:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971521] pcieport 0000:00:16.6: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971641] pcie_pme 0000:00:16.6:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971741] pcieport 0000:00:16.7: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971862] pcie_pme 0000:00:16.7:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.971969] pcieport 0000:00:17.0: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.972089] pcie_pme 0000:00:17.0:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.972180] pcieport 0000:00:17.1: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.972292] pcie_pme 0000:00:17.1:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.972389] pcieport 0000:00:17.2: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.972552] pcie_pme 0000:00:17.2:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.972697] pcieport 0000:00:17.3: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.972861] pcie_pme 0000:00:17.3:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.973007] pcieport 0000:00:17.4: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.973197] pcie_pme 0000:00:17.4:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.973350] pcieport 0000:00:17.5: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.973510] pcie_pme 0000:00:17.5:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.973653] pcieport 0000:00:17.6: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.973817] pcie_pme 0000:00:17.6:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.973963] pcieport 0000:00:17.7: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.974129] pcie_pme 0000:00:17.7:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.974276] pcieport 0000:00:18.0: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.974435] pcie_pme 0000:00:18.0:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.974577] pcieport 0000:00:18.1: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.974738] pcie_pme 0000:00:18.1:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.974878] pcieport 0000:00:18.2: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.975044] pcie_pme 0000:00:18.2:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.975191] pcieport 0000:00:18.3: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.975354] pcie_pme 0000:00:18.3:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.975493] pcieport 0000:00:18.4: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.975668] pcie_pme 0000:00:18.4:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.975813] pcieport 0000:00:18.5: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.975969] pcie_pme 0000:00:18.5:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.976116] pcieport 0000:00:18.6: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.976282] pcie_pme 0000:00:18.6:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.976398] pcieport 0000:00:18.7: Signaling PME through PCIe PME interrupt Mar 10 09:26:55 lustre-utils01 kernel: [ 0.976513] pcie_pme 0000:00:18.7:pcie01: service driver pcie_pme loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.976536] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.977137] pciehp 0000:00:15.0:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.979345] pciehp 0000:00:15.0:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.979402] pciehp 0000:00:15.1:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.979583] pciehp 0000:00:15.1:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.979636] pciehp 0000:00:15.2:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.979821] pciehp 0000:00:15.2:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.979931] pciehp 0000:00:15.3:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.980129] pciehp 0000:00:15.3:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.980192] pciehp 0000:00:15.4:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.980385] pciehp 0000:00:15.4:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.980439] pciehp 0000:00:15.5:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.980616] pciehp 0000:00:15.5:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.980669] pciehp 0000:00:15.6:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.980849] pciehp 0000:00:15.6:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.980909] pciehp 0000:00:15.7:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.981087] pciehp 0000:00:15.7:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.981163] pciehp 0000:00:16.0:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.981353] pciehp 0000:00:16.0:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.981408] pciehp 0000:00:16.1:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.981839] pciehp 0000:00:16.1:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.981920] pciehp 0000:00:16.2:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.982182] pciehp 0000:00:16.2:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.982269] pciehp 0000:00:16.3:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.982532] pciehp 0000:00:16.3:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.982617] pciehp 0000:00:16.4:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.982887] pciehp 0000:00:16.4:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.982968] pciehp 0000:00:16.5:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.983229] pciehp 0000:00:16.5:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.983314] pciehp 0000:00:16.6:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.983576] pciehp 0000:00:16.6:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.983663] pciehp 0000:00:16.7:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.983920] pciehp 0000:00:16.7:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.984005] pciehp 0000:00:17.0:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.984815] pciehp 0000:00:17.0:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.984882] pciehp 0000:00:17.1:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.985160] pciehp 0000:00:17.1:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.985246] pciehp 0000:00:17.2:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.985510] pciehp 0000:00:17.2:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.985597] pciehp 0000:00:17.3:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.985862] pciehp 0000:00:17.3:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.985947] pciehp 0000:00:17.4:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.986211] pciehp 0000:00:17.4:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.986299] pciehp 0000:00:17.5:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.986563] pciehp 0000:00:17.5:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.986648] pciehp 0000:00:17.6:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.986910] pciehp 0000:00:17.6:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.986999] pciehp 0000:00:17.7:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.987193] pciehp 0000:00:17.7:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.987249] pciehp 0000:00:18.0:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.987432] pciehp 0000:00:18.0:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.987486] pciehp 0000:00:18.1:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.987663] pciehp 0000:00:18.1:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.987717] pciehp 0000:00:18.2:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.987916] pciehp 0000:00:18.2:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.987971] pciehp 0000:00:18.3:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.988180] pciehp 0000:00:18.3:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.988238] pciehp 0000:00:18.4:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.988417] pciehp 0000:00:18.4:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.988472] pciehp 0000:00:18.5:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.988666] pciehp 0000:00:18.5:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.988721] pciehp 0000:00:18.6:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.988915] pciehp 0000:00:18.6:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.988972] pciehp 0000:00:18.7:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 15ad ss_did 7a0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.989183] pciehp 0000:00:18.7:pcie04: service driver pciehp loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 0.989190] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.989338] intel_idle: does not run on family 6 model 26 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.989541] ACPI: AC Adapter [ACAD] (on-line) Mar 10 09:26:55 lustre-utils01 kernel: [ 0.989802] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 Mar 10 09:26:55 lustre-utils01 kernel: [ 0.989938] ACPI: Power Button [PWRF] Mar 10 09:26:55 lustre-utils01 kernel: [ 1.040103] GHES: HEST is not enabled! Mar 10 09:26:55 lustre-utils01 kernel: [ 1.040269] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled Mar 10 09:26:55 lustre-utils01 kernel: [ 1.067168] 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Mar 10 09:26:55 lustre-utils01 kernel: [ 1.094661] 00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Mar 10 09:26:55 lustre-utils01 kernel: [ 1.095926] Linux agpgart interface v0.103 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.096090] agpgart-intel 0000:00:00.0: Intel 440BX Chipset Mar 10 09:26:55 lustre-utils01 kernel: [ 1.096876] agpgart-intel 0000:00:00.0: AGP aperture is 256M @ 0x0 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.098067] brd: module loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 1.098686] loop: module loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 1.098878] ata_piix 0000:00:07.1: version 2.13 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.099233] scsi0 : ata_piix Mar 10 09:26:55 lustre-utils01 kernel: [ 1.099362] scsi1 : ata_piix Mar 10 09:26:55 lustre-utils01 kernel: [ 1.099455] ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0x10c0 irq 14 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.099562] ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0x10c8 irq 15 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.099931] libphy: Fixed MDIO Bus: probed Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100057] tun: Universal TUN/TAP device driver, 1.6 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100140] tun: (C) 1999-2004 Max Krasnyansky Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100290] PPP generic driver version 2.4.2 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100408] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100505] ehci-pci: EHCI PCI platform driver Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100595] ehci-platform: EHCI generic platform driver Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100691] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100801] uhci_hcd: USB Universal Host Controller Interface driver Mar 10 09:26:55 lustre-utils01 kernel: [ 1.100963] i8042: PNP: PS/2 Controller [PNP0303:KBC,PNP0f13:MOUS] at 0x60,0x64 irq 1,12 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.102270] serio: i8042 KBD port at 0x60,0x64 irq 1 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.102361] serio: i8042 AUX port at 0x60,0x64 irq 12 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.102525] mousedev: PS/2 mouse device common for all mice Mar 10 09:26:55 lustre-utils01 kernel: [ 1.103207] rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.103522] rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs Mar 10 09:26:55 lustre-utils01 kernel: [ 1.103846] device-mapper: uevent: version 1.0.3 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.104042] device-mapper: ioctl: 4.23.1-ioctl (2012-12-18) initialised: dm-devel@redhat.com Mar 10 09:26:55 lustre-utils01 kernel: [ 1.104277] cpuidle: using governor ladder Mar 10 09:26:55 lustre-utils01 kernel: [ 1.104361] cpuidle: using governor menu Mar 10 09:26:55 lustre-utils01 kernel: [ 1.104665] ledtrig-cpu: registered to indicate activity on CPUs Mar 10 09:26:55 lustre-utils01 kernel: [ 1.104793] EFI Variables Facility v0.08 2004-May-17 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.104963] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.105428] ashmem: initialized Mar 10 09:26:55 lustre-utils01 kernel: [ 1.105615] TCP: cubic registered Mar 10 09:26:55 lustre-utils01 kernel: [ 1.105763] NET: Registered protocol family 10 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.106012] NET: Registered protocol family 17 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.106101] Key type dns_resolver registered Mar 10 09:26:55 lustre-utils01 kernel: [ 1.106346] Loading module verification certificates Mar 10 09:26:55 lustre-utils01 kernel: [ 1.109005] MODSIGN: Loaded cert 'Magrathea: Glacier signing key: 8e2f2f6cc80151e84c7600385debfe328f065719' Mar 10 09:26:55 lustre-utils01 kernel: [ 1.109266] registered taskstats version 1 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.111569] Key type trusted registered Mar 10 09:26:55 lustre-utils01 kernel: [ 1.113210] Key type encrypted registered Mar 10 09:26:55 lustre-utils01 kernel: [ 1.118951] rtc_cmos 00:03: setting system clock to 2014-03-10 09:26:46 UTC (1394443606) Mar 10 09:26:55 lustre-utils01 kernel: [ 1.119202] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found Mar 10 09:26:55 lustre-utils01 kernel: [ 1.119295] EDD information not available. Mar 10 09:26:55 lustre-utils01 kernel: [ 1.297125] ata2.00: ATAPI: VMware Virtual IDE CDROM Drive, 00000001, max UDMA/33 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.313189] ata2.00: configured for UDMA/33 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.314246] scsi 1:0:0:0: CD-ROM NECVMWar VMware IDE CDR10 1.00 PQ: 0 ANSI: 5 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.315594] sr0: scsi3-mmc drive: 1x/1x writer dvd-ram cd/rw xa/form2 cdda tray Mar 10 09:26:55 lustre-utils01 kernel: [ 1.315733] cdrom: Uniform CD-ROM driver Revision: 3.20 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.315906] sr 1:0:0:0: Attached scsi CD-ROM sr0 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.315945] sr 1:0:0:0: Attached scsi generic sg0 type 5 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.318207] Freeing unused kernel memory: 1016k freed Mar 10 09:26:55 lustre-utils01 kernel: [ 1.318561] Write protecting the kernel read-only data: 12288k Mar 10 09:26:55 lustre-utils01 kernel: [ 1.322561] Freeing unused kernel memory: 1000k freed Mar 10 09:26:55 lustre-utils01 kernel: [ 1.328459] Freeing unused kernel memory: 952k freed Mar 10 09:26:55 lustre-utils01 kernel: [ 1.370316] Disabling lock debugging due to kernel taint Mar 10 09:26:55 lustre-utils01 kernel: [ 1.380349] VMware vmxnet3 virtual NIC driver - version 1.1.29.0-k-NAPI Mar 10 09:26:55 lustre-utils01 kernel: [ 1.392362] # of Tx queues : 1, # of Rx queues : 1 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.393479] vmxnet3 0000:0b:00.0 (unregistered net_device): features: sg csum vlan jf tso tsoIPv6 lro highDMA Mar 10 09:26:55 lustre-utils01 kernel: [ 1.407972] vmxnet3 0000:0b:00.0: irq 72 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 1.408053] vmxnet3 0000:0b:00.0: irq 73 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 1.408570] eth0: NIC Link is Up 10000 Mbps Mar 10 09:26:55 lustre-utils01 kernel: [ 1.423576] VMware PVSCSI driver - version 1.0.2.0-k Mar 10 09:26:55 lustre-utils01 kernel: [ 1.424506] vmw_pvscsi: using 64bit dma Mar 10 09:26:55 lustre-utils01 kernel: [ 1.428333] vmw_pvscsi 0000:03:00.0: vmw_pvscsi: host->max_id: 16 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.429677] vmw_pvscsi 0000:03:00.0: irq 74 for MSI/MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 1.429814] vmw_pvscsi: using MSI-X Mar 10 09:26:55 lustre-utils01 kernel: [ 1.429983] scsi2 : VMware PVSCSI storage adapter rev 2, req/cmp/msg rings: 8/8/1 pages, cmd_per_lun=64 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.430190] vmw_pvscsi 0000:03:00.0: VMware PVSCSI rev 2 host #2 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.430421] scsi 2:0:0:0: Direct-Access VMware Virtual disk 1.0 PQ: 0 ANSI: 2 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.430618] scsi 2:0:1:0: Direct-Access VMware Virtual disk 1.0 PQ: 0 ANSI: 2 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.430848] scsi 2:0:2:0: Direct-Access VMware Virtual disk 1.0 PQ: 0 ANSI: 2 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.431564] sd 2:0:0:0: [sda] 33554432 512-byte logical blocks: (17.1 GB/16.0 GiB) Mar 10 09:26:55 lustre-utils01 kernel: [ 1.431724] sd 2:0:0:0: [sda] Write Protect is off Mar 10 09:26:55 lustre-utils01 kernel: [ 1.431820] sd 2:0:0:0: [sda] Mode Sense: 61 00 00 00 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.431854] sd 2:0:0:0: [sda] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.431967] sd 2:0:0:0: [sda] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.432288] sd 2:0:0:0: [sda] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.432400] sd 2:0:0:0: [sda] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.432712] sd 2:0:0:0: Attached scsi generic sg1 type 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.433014] sd 2:0:1:0: [sdb] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB) Mar 10 09:26:55 lustre-utils01 kernel: [ 1.433185] sd 2:0:1:0: [sdb] Write Protect is off Mar 10 09:26:55 lustre-utils01 kernel: [ 1.433283] sd 2:0:1:0: [sdb] Mode Sense: 61 00 00 00 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.433308] sd 2:0:1:0: [sdb] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.433408] sd 2:0:1:0: [sdb] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.433682] sd 2:0:1:0: [sdb] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.433772] sd 2:0:1:0: [sdb] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.434380] sd 2:0:1:0: Attached scsi generic sg2 type 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.434750] sd 2:0:2:0: Attached scsi generic sg3 type 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.434985] sd 2:0:2:0: [sdc] 67108864 512-byte logical blocks: (34.3 GB/32.0 GiB) Mar 10 09:26:55 lustre-utils01 kernel: [ 1.435183] sd 2:0:2:0: [sdc] Write Protect is off Mar 10 09:26:55 lustre-utils01 kernel: [ 1.435343] sd 2:0:2:0: [sdc] Mode Sense: 61 00 00 00 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.435377] sd 2:0:2:0: [sdc] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.435491] sd 2:0:2:0: [sdc] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.435768] sda: sda1 sda2 < sda5 > Mar 10 09:26:55 lustre-utils01 kernel: [ 1.436214] sd 2:0:0:0: [sda] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.436327] sd 2:0:0:0: [sda] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.436485] sd 2:0:0:0: [sda] Attached SCSI disk Mar 10 09:26:55 lustre-utils01 kernel: [ 1.436598] sdb: unknown partition table Mar 10 09:26:55 lustre-utils01 kernel: [ 1.436935] sd 2:0:1:0: [sdb] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.437075] sd 2:0:1:0: [sdb] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.437233] sd 2:0:1:0: [sdb] Attached SCSI disk Mar 10 09:26:55 lustre-utils01 kernel: [ 1.437516] sd 2:0:2:0: [sdc] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.439635] sd 2:0:2:0: [sdc] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.443690] sdc: unknown partition table Mar 10 09:26:55 lustre-utils01 kernel: [ 1.444024] sd 2:0:2:0: [sdc] Cache data unavailable Mar 10 09:26:55 lustre-utils01 kernel: [ 1.444137] sd 2:0:2:0: [sdc] Assuming drive cache: write through Mar 10 09:26:55 lustre-utils01 kernel: [ 1.444297] sd 2:0:2:0: [sdc] Attached SCSI disk Mar 10 09:26:55 lustre-utils01 kernel: [ 1.532999] Floppy drive(s): fd0 is 1.44M Mar 10 09:26:55 lustre-utils01 kernel: [ 1.550207] FDC 0 is a post-1991 82077 Mar 10 09:26:55 lustre-utils01 kernel: [ 1.843045] bio: create slab at 1 Mar 10 09:26:55 lustre-utils01 kernel: [ 2.285645] EXT4-fs (dm-0): INFO: recovery required on readonly filesystem Mar 10 09:26:55 lustre-utils01 kernel: [ 2.285767] EXT4-fs (dm-0): write access will be enabled during recovery Mar 10 09:26:55 lustre-utils01 kernel: [ 3.776510] EXT4-fs (dm-0): orphan cleanup on readonly fs Mar 10 09:26:55 lustre-utils01 kernel: [ 3.786524] EXT4-fs (dm-0): ext4_orphan_cleanup: deleting unreferenced inode 524707 Mar 10 09:26:55 lustre-utils01 kernel: [ 3.786609] EXT4-fs (dm-0): 1 orphan inode deleted Mar 10 09:26:55 lustre-utils01 kernel: [ 3.786767] EXT4-fs (dm-0): recovery complete Mar 10 09:26:55 lustre-utils01 kernel: [ 3.792491] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) Mar 10 09:26:55 lustre-utils01 kernel: [ 5.830343] Adding 1048572k swap on /dev/sdb. Priority:-1 extents:1 across:1048572k Mar 10 09:26:55 lustre-utils01 kernel: [ 5.836365] Adding 33554428k swap on /dev/sdc. Priority:-2 extents:1 across:33554428k Mar 10 09:26:55 lustre-utils01 kernel: [ 5.866964] Adding 4190204k swap on /dev/mapper/precise--dev64-swap_1. Priority:-3 extents:1 across:4190204k Mar 10 09:26:55 lustre-utils01 kernel: [ 5.997532] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Mar 10 09:26:55 lustre-utils01 kernel: [ 6.292989] EXT4-fs (dm-0): re-mounted. Opts: errors=panic Mar 10 09:26:55 lustre-utils01 kernel: [ 6.567474] lp: driver loaded but no devices found Mar 10 09:26:55 lustre-utils01 kernel: [ 7.082371] parport_pc 00:08: reported by Plug and Play ACPI Mar 10 09:26:55 lustre-utils01 kernel: [ 7.082599] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE] Mar 10 09:26:55 lustre-utils01 kernel: [ 7.126095] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Mar 10 09:26:55 lustre-utils01 kernel: [ 7.174302] lp0: using parport0 (interrupt-driven). Mar 10 09:26:55 lustre-utils01 kernel: [ 7.207854] piix4_smbus 0000:00:07.3: Host SMBus controller not enabled! Mar 10 09:26:55 lustre-utils01 kernel: [ 7.331879] site: intr type 3, mode 0, 2 vectors allocated Mar 10 09:26:55 lustre-utils01 kernel: [ 7.333024] site: NIC Link is Up 10000 Mbps Mar 10 09:26:55 lustre-utils01 kernel: [ 7.632926] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input2 Mar 10 09:26:55 lustre-utils01 kernel: [ 7.850410] RPC: Registered named UNIX socket transport module. Mar 10 09:26:55 lustre-utils01 kernel: [ 7.850413] RPC: Registered udp transport module. Mar 10 09:26:55 lustre-utils01 kernel: [ 7.850414] RPC: Registered tcp transport module. Mar 10 09:26:55 lustre-utils01 kernel: [ 7.850415] RPC: Registered tcp NFSv4.1 backchannel transport module. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.072118] FS-Cache: Loaded Mar 10 09:26:55 lustre-utils01 kernel: [ 8.142509] FS-Cache: Netfs 'nfs' registered for caching Mar 10 09:26:55 lustre-utils01 kernel: [ 8.512573] microcode: CPU0 sig=0x106a4, pf=0x1, revision=0x19 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.556590] microcode: Microcode Update Driver: v2.00 , Peter Oruba Mar 10 09:26:55 lustre-utils01 kernel: [ 8.559769] ppdev: user-space parallel port driver Mar 10 09:26:55 lustre-utils01 kernel: [ 8.567195] [drm] Initialized drm 1.1.0 20060810 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680278] [drm] Capabilities: Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680281] [drm] Rect copy. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680282] [drm] Cursor. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680283] [drm] Cursor bypass. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680283] [drm] Cursor bypass 2. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680284] [drm] 8bit emulation. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680285] [drm] Alpha cursor. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680285] [drm] Extended Fifo. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680286] [drm] Multimon. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680286] [drm] Pitchlock. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680287] [drm] Irq mask. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680288] [drm] Display Topology. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680288] [drm] GMR. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680289] [drm] Traces. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680290] [drm] GMR2. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680290] [drm] Screen Object 2. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680291] [drm] Max GMR ids is 64 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680292] [drm] Max GMR descriptors is 4096 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680293] [drm] Max number of GMR pages is 65536 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680294] [drm] Max dedicated hypervisor surface memory is 163840 kiB Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680295] [drm] VRAM at 0xd8000000 size is 4096 kiB Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680296] [drm] MMIO at 0xd0800000 size is 256 kiB Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680297] [drm] global init. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680338] [TTM] Zone kernel: Available graphics memory: 8217742 kiB Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680340] [TTM] Zone dma32: Available graphics memory: 2097152 kiB Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680341] [TTM] Initializing pool allocator Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680344] [TTM] Initializing DMA pool allocator Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680646] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010). Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680648] [drm] No driver support for vblank timestamp query. Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680711] [drm] Screen objects system initialized Mar 10 09:26:55 lustre-utils01 kernel: [ 8.680716] [drm] Initialized vmwgfx 2.4.0 20120209 for 0000:00:0f.0 on minor 0 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.951788] [drm] width 640 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.951809] [drm] height 480 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.951822] [drm] bpp 32 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.954468] [drm] Fifo max 0x00040000 min 0x00001000 cap 0x0000077f Mar 10 09:26:55 lustre-utils01 kernel: [ 8.957665] [drm] width 640 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.957676] [drm] height 480 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.957685] [drm] bpp 32 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.959714] [drm] Fifo max 0x00040000 min 0x00001000 cap 0x0000077f Mar 10 09:26:55 lustre-utils01 kernel: [ 8.966982] [drm] width 640 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.967008] [drm] height 480 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.967032] [drm] bpp 32 Mar 10 09:26:55 lustre-utils01 kernel: [ 8.968969] [drm] Fifo max 0x00040000 min 0x00001000 cap 0x0000077f Mar 10 09:26:55 lustre-utils01 kernel: [ 9.573180] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Mar 10 09:26:57 lustre-utils01 kernel: [ 11.688733] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689113] acpiphp: Slot [32] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689140] acpiphp: Slot [33] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689160] acpiphp: Slot [34] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689181] acpiphp: Slot [35] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689202] acpiphp: Slot [36] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689226] acpiphp: Slot [37] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689247] acpiphp: Slot [38] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689268] acpiphp: Slot [39] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689293] acpiphp: Slot [40] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689317] acpiphp: Slot [41] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689339] acpiphp: Slot [42] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689361] acpiphp: Slot [43] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689383] acpiphp: Slot [44] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689405] acpiphp: Slot [45] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689426] acpiphp: Slot [46] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689447] acpiphp: Slot [47] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689467] acpiphp: Slot [48] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689491] acpiphp: Slot [49] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689516] acpiphp: Slot [50] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689536] acpiphp: Slot [51] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689557] acpiphp: Slot [52] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689580] acpiphp: Slot [53] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689601] acpiphp: Slot [54] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689622] acpiphp: Slot [55] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689643] acpiphp: Slot [56] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689667] acpiphp: Slot [57] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689689] acpiphp: Slot [58] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689712] acpiphp: Slot [59] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689733] acpiphp: Slot [60] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689759] acpiphp: Slot [61] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689780] acpiphp: Slot [62] registered Mar 10 09:26:57 lustre-utils01 kernel: [ 11.689802] acpiphp: Slot [63] registered Mar 10 09:27:00 lustre-utils01 kernel: [ 14.776780] Bridge firewalling registered Mar 10 09:27:00 lustre-utils01 kernel: [ 14.815941] 8021q: 802.1Q VLAN Support v1.8 Mar 10 09:27:00 lustre-utils01 kernel: [ 14.815956] 8021q: adding VLAN 0 to HW filter on device site Mar 10 09:27:01 lustre-utils01 kernel: [ 16.501966] ipmi message handler version 39.2 Mar 10 09:27:01 lustre-utils01 kernel: [ 16.510726] IPMI System Interface driver. Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512600] ipmi_si: Adding default-specified kcs state machine Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512605] ipmi_si: Trying default-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512644] ipmi_si: Interface detection failed Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512649] ipmi_si: Adding default-specified smic state machine Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512652] ipmi_si: Trying default-specified smic state machine at i/o address 0xca9, slave address 0x0, irq 0 Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512663] ipmi_si: Interface detection failed Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512666] ipmi_si: Adding default-specified bt state machine Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512669] ipmi_si: Trying default-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0 Mar 10 09:27:01 lustre-utils01 kernel: [ 16.512707] ipmi_si: Interface detection failed Mar 10 09:27:01 lustre-utils01 kernel: [ 16.515408] ipmi_si: Unable to find any System Interface(s) Mar 10 09:27:02 lustre-utils01 kernel: [ 17.171289] LNet: HW CPU cores: 1, npartitions: 1 Mar 10 09:27:10 lustre-utils01 kernel: [ 25.306004] Lustre: Lustre: Build Version: v2_5_0_0--CHANGED-3.8.0-37-generic Mar 10 09:27:10 lustre-utils01 kernel: [ 25.416533] LNet: Added LNI 172.17.137.75@tcp [8/256/0/180] Mar 10 09:27:10 lustre-utils01 kernel: [ 25.420669] LNet: Accept secure, port 988 Mar 10 09:27:10 lustre-utils01 kernel: [ 25.904052] Lustre: Mounted lus08-client Mar 10 09:27:11 lustre-utils01 kernel: [ 26.102686] Lustre: Mounted lus13-client Mar 10 09:27:11 lustre-utils01 kernel: [ 26.687998] Lustre: Mounted lus09-client Mar 10 09:27:11 lustre-utils01 kernel: [ 26.688003] Lustre: Skipped 1 previous similar message Mar 10 09:27:12 lustre-utils01 kernel: [ 27.688182] Lustre: Layout lock feature supported. Mar 10 09:27:12 lustre-utils01 kernel: [ 27.930362] Lustre: Mounted lus11-client Mar 10 09:27:12 lustre-utils01 kernel: [ 27.930367] Lustre: Skipped 3 previous similar messages Mar 10 09:27:13 lustre-utils01 kernel: [ 27.997238] Lustre: Layout lock feature supported. Mar 10 09:28:55 lustre-utils01 kernel: [ 130.331136] Lustre: 1999:0:(client.c:1897:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394443630/real 1394443630] req@ffff88042883a800 x1462180123770960/t0(0) o8->lus08-OST0000-osc-ffff880428f17800@172.17.31.18@tcp:28/4 lens 400/544 e 0 to 1 dl 1394443735 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1