Jun 16 10:00:08 atlas-mds1 kernel: [515727.290363] ptlrpcd_18: page allocation failure. order:1, mode:0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290537] ptlrpcd_4: page allocation failure. order:1, mode:0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290541] Pid: 12116, comm: ptlrpcd_4 Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290543] Call Trace: Jun 16 10:00:08 atlas-mds1 kernel: [515727.290553] [] ? __alloc_pages_nodemask+0x757/0x8d0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290558] [] ? kmem_getpages+0x62/0x170 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290561] [] ? fallback_alloc+0x1ba/0x270 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290564] [] ? cache_grow+0x2cf/0x320 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290567] ptlrpcd_12: page allocation failure. order:1, mode:0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290570] [] ? ____cache_alloc_node+0x99/0x160 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290572] Pid: 12124, comm: ptlrpcd_12 Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290574] Call Trace: Jun 16 10:00:08 atlas-mds1 kernel: [515727.290579] [] ? __alloc_pages_nodemask+0x757/0x8d0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290593] [] ? cfs_alloc+0x30/0x60 [libcfs] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290595] [] ? kmem_getpages+0x62/0x170 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290598] [] ? __kmalloc+0x189/0x220 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290600] [] ? fallback_alloc+0x1ba/0x270 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290608] [] ? cfs_alloc+0x30/0x60 [libcfs] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290610] [] ? cache_grow+0x2cf/0x320 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290614] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290616] [] ? ____cache_alloc_node+0x99/0x160 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290619] [] ? lprocfs_counter_add+0x156/0x182 [lvfs] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290631] [] ? cfs_alloc+0x30/0x60 [libcfs] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290669] [] ? ptl_send_rpc+0x49e/0xc40 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290672] [] ? __kmalloc+0x189/0x220 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290700] [] ? ptlrpc_send_new_req+0x45b/0x7a0 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290709] [] ? cfs_alloc+0x30/0x60 [libcfs] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290714] [] ? lock_timer_base+0x3c/0x70 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290717] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290747] [] ? ptlrpc_check_set+0x878/0x1b20 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290750] [] ? lprocfs_counter_add+0x156/0x182 [lvfs] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290754] [] ? del_timer_sync+0x22/0x30 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290790] [] ? after_reply+0x32a/0xd90 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290823] [] ? ptlrpcd_check+0x53b/0x560 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290827] [] ? lock_timer_base+0x3c/0x70 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290858] [] ? ptlrpcd+0x233/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290887] [] ? ptlrpc_check_set+0xfd1/0x1b20 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290892] [] ? default_wake_function+0x0/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290922] [] ? ptlrpcd_check+0x53b/0x560 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290953] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290981] [] ? ptlrpcd+0x233/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.290985] [] ? child_rip+0xa/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.290988] [] ? default_wake_function+0x0/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291018] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291046] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291075] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291078] [] ? child_rip+0xa/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291080] [] ? child_rip+0x0/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291082] Mem-Info: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291109] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291111] Node 0 DMA per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291139] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291142] CPU 0: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291144] CPU 1: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291146] [] ? child_rip+0x0/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291148] CPU 2: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291150] Mem-Info: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291151] CPU 3: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291153] Node 0 CPU 4: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291155] DMA per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291157] CPU 5: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291159] CPU 0: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291160] CPU 6: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291162] CPU 1: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291164] CPU 7: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291166] CPU 2: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291167] CPU 8: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291169] CPU 3: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291171] CPU 9: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291173] CPU 4: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291174] CPU 10: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291176] CPU 5: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291178] CPU 11: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291179] CPU 6: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291181] CPU 12: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291183] CPU 7: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291184] CPU 13: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291186] CPU 8: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291188] CPU 14: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291190] CPU 9: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291191] CPU 15: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291193] CPU 10: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291195] CPU 16: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291197] CPU 11: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291198] CPU 17: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291200] CPU 12: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291202] CPU 18: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291204] CPU 13: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291205] CPU 19: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291208] CPU 14: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291209] CPU 20: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291211] CPU 15: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291213] CPU 21: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291215] CPU 16: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291216] CPU 22: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291218] CPU 17: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291220] CPU 23: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291222] CPU 18: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291223] Node 0 CPU 19: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291226] DMA32 per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291227] CPU 20: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291229] CPU 0: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291231] CPU 21: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291232] CPU 1: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291234] CPU 22: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291236] CPU 2: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291238] CPU 23: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291240] CPU 3: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291241] Node 0 CPU 4: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291243] DMA32 per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291245] CPU 5: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291247] CPU 0: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291249] CPU 6: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291251] CPU 1: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291253] CPU 7: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291255] CPU 2: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291256] CPU 8: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291258] CPU 3: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291260] CPU 9: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291262] CPU 4: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291263] CPU 10: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291265] CPU 5: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291267] CPU 11: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291268] CPU 6: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291270] CPU 12: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291272] CPU 7: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291274] CPU 13: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291276] CPU 8: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291277] CPU 14: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291279] CPU 9: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291281] CPU 15: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291283] CPU 10: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291285] CPU 16: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291287] CPU 11: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291289] CPU 17: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291290] CPU 12: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291292] CPU 18: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291293] CPU 13: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291295] CPU 19: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291296] CPU 14: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291298] CPU 20: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291299] CPU 15: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291301] CPU 21: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291303] CPU 16: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291304] CPU 22: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291306] CPU 17: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291307] CPU 23: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291309] CPU 18: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291310] Node 0 CPU 19: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291312] Normal per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291313] CPU 20: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291315] CPU 0: hi: 186, btch: 31 usd: 89 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291317] CPU 21: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291319] CPU 1: hi: 186, btch: 31 usd: 23 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291320] CPU 22: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291322] CPU 2: hi: 186, btch: 31 usd: 155 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291324] CPU 23: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291326] CPU 3: hi: 186, btch: 31 usd: 45 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291327] Node 0 CPU 4: hi: 186, btch: 31 usd: 108 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291329] Normal per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291331] CPU 5: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291332] CPU 0: hi: 186, btch: 31 usd: 89 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291334] CPU 6: hi: 186, btch: 31 usd: 156 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291336] CPU 1: hi: 186, btch: 31 usd: 23 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291338] CPU 7: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291340] CPU 2: hi: 186, btch: 31 usd: 155 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291342] CPU 8: hi: 186, btch: 31 usd: 33 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291344] CPU 3: hi: 186, btch: 31 usd: 45 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291345] CPU 9: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291347] CPU 4: hi: 186, btch: 31 usd: 108 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291349] CPU 10: hi: 186, btch: 31 usd: 17 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291351] CPU 5: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291353] CPU 11: hi: 186, btch: 31 usd: 9 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291355] CPU 6: hi: 186, btch: 31 usd: 156 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291356] CPU 12: hi: 186, btch: 31 usd: 35 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291358] CPU 7: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291360] CPU 13: hi: 186, btch: 31 usd: 3 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291362] CPU 8: hi: 186, btch: 31 usd: 33 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291363] CPU 14: hi: 186, btch: 31 usd: 140 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291365] CPU 9: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291367] CPU 15: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291369] CPU 10: hi: 186, btch: 31 usd: 17 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291371] CPU 16: hi: 186, btch: 31 usd: 135 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291373] CPU 11: hi: 186, btch: 31 usd: 9 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291374] CPU 17: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291376] CPU 12: hi: 186, btch: 31 usd: 35 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291378] CPU 18: hi: 186, btch: 31 usd: 129 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291380] CPU 13: hi: 186, btch: 31 usd: 3 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291382] CPU 19: hi: 186, btch: 31 usd: 17 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291383] CPU 14: hi: 186, btch: 31 usd: 140 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291385] CPU 20: hi: 186, btch: 31 usd: 54 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291387] CPU 15: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291388] CPU 21: hi: 186, btch: 31 usd: 12 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291390] CPU 16: hi: 186, btch: 31 usd: 135 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291392] CPU 22: hi: 186, btch: 31 usd: 47 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291394] CPU 17: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291395] CPU 23: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291397] CPU 18: hi: 186, btch: 31 usd: 129 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291399] Node 1 CPU 19: hi: 186, btch: 31 usd: 17 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291401] Normal per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291403] CPU 20: hi: 186, btch: 31 usd: 54 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291405] CPU 0: hi: 186, btch: 31 usd: 23 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291406] CPU 21: hi: 186, btch: 31 usd: 12 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291408] CPU 1: hi: 186, btch: 31 usd: 120 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291409] CPU 22: hi: 186, btch: 31 usd: 47 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291411] CPU 2: hi: 186, btch: 31 usd: 64 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291412] CPU 23: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291414] CPU 3: hi: 186, btch: 31 usd: 47 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291415] Node 1 CPU 4: hi: 186, btch: 31 usd: 2 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291417] Normal per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515727.291418] CPU 5: hi: 186, btch: 31 usd: 38 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291419] CPU 0: hi: 186, btch: 31 usd: 23 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291421] CPU 6: hi: 186, btch: 31 usd: 3 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291422] CPU 1: hi: 186, btch: 31 usd: 120 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291424] CPU 7: hi: 186, btch: 31 usd: 49 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291425] CPU 2: hi: 186, btch: 31 usd: 64 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291427] CPU 8: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291428] CPU 3: hi: 186, btch: 31 usd: 47 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291429] CPU 9: hi: 186, btch: 31 usd: 73 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291431] CPU 4: hi: 186, btch: 31 usd: 2 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291432] CPU 10: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291434] CPU 5: hi: 186, btch: 31 usd: 38 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291435] CPU 11: hi: 186, btch: 31 usd: 124 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291436] CPU 6: hi: 186, btch: 31 usd: 3 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291438] CPU 12: hi: 186, btch: 31 usd: 13 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291439] CPU 7: hi: 186, btch: 31 usd: 49 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291441] CPU 13: hi: 186, btch: 31 usd: 148 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291442] CPU 8: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291443] CPU 14: hi: 186, btch: 31 usd: 53 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291445] CPU 9: hi: 186, btch: 31 usd: 72 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291446] CPU 15: hi: 186, btch: 31 usd: 126 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291448] CPU 10: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291449] CPU 16: hi: 186, btch: 31 usd: 12 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291450] CPU 11: hi: 186, btch: 31 usd: 124 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291452] CPU 17: hi: 186, btch: 31 usd: 38 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291453] CPU 12: hi: 186, btch: 31 usd: 13 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291454] CPU 18: hi: 186, btch: 31 usd: 18 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291456] CPU 13: hi: 186, btch: 31 usd: 148 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291457] CPU 19: hi: 186, btch: 31 usd: 116 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291459] CPU 14: hi: 186, btch: 31 usd: 53 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291460] CPU 20: hi: 186, btch: 31 usd: 11 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291462] CPU 15: hi: 186, btch: 31 usd: 126 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291463] CPU 21: hi: 186, btch: 31 usd: 142 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291465] CPU 16: hi: 186, btch: 31 usd: 12 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291466] CPU 22: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291467] CPU 17: hi: 186, btch: 31 usd: 38 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291469] CPU 23: hi: 186, btch: 31 usd: 87 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291470] CPU 18: hi: 186, btch: 31 usd: 18 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291474] active_anon:10205060 inactive_anon:86396 isolated_anon:0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291475] active_file:11640566 inactive_file:29949329 isolated_file:0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291475] unevictable:8694 dirty:1198 writeback:520 unstable:0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291476] free:143641 slab_reclaimable:2083795 slab_unreclaimable:11110797 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291477] mapped:9355 shmem:153220 pagetables:21028 bounce:0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291478] CPU 19: hi: 186, btch: 31 usd: 116 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291479] Node 0 CPU 20: hi: 186, btch: 31 usd: 11 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291484] DMA free:15740kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jun 16 10:00:08 atlas-mds1 kernel: [515727.291487] CPU 21: hi: 186, btch: 31 usd: 142 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291488] lowmem_reserve[]:CPU 22: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291490] 0CPU 23: hi: 186, btch: 31 usd: 91 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291492] 1931active_anon:10205060 inactive_anon:86396 isolated_anon:0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291496] active_file:11640566 inactive_file:29949329 isolated_file:0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291497] unevictable:8694 dirty:1198 writeback:520 unstable:0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291498] free:143641 slab_reclaimable:2083795 slab_unreclaimable:11110797 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291499] mapped:9355 shmem:153220 pagetables:21028 bounce:0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291500] 129191Node 0 129191DMA free:15740kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jun 16 10:00:08 atlas-mds1 kernel: [515727.291508] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291509] lowmem_reserve[]:Node 0 0DMA32 free:386760kB min:976kB low:1220kB high:1464kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1977504kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jun 16 10:00:08 atlas-mds1 kernel: [515727.291516] 1931lowmem_reserve[]: 129191 0 129191 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291519] 127260Node 0 127260DMA32 free:386760kB min:976kB low:1220kB high:1464kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1977504kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jun 16 10:00:08 atlas-mds1 kernel: [515727.291527] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291527] lowmem_reserve[]:Node 0 0Normal free:88428kB min:64528kB low:80660kB high:96792kB active_anon:28039004kB inactive_anon:281040kB active_file:21150504kB inactive_file:56891400kB unevictable:10916kB isolated(anon):0kB isolated(file):0kB present:130314240kB mlocked:0kB dirty:1312kB writeback:0kB mapped:34904kB shmem:461336kB slab_reclaimable:2840868kB slab_unreclaimable:20564596kB kernel_stack:23512kB pagetables:63984kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jun 16 10:00:08 atlas-mds1 kernel: [515727.291535] 0lowmem_reserve[]: 127260 0 127260 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291538] 0Node 0 0Normal free:88428kB min:64528kB low:80660kB high:96792kB active_anon:28039004kB inactive_anon:281040kB active_file:21150504kB inactive_file:56891400kB unevictable:10916kB isolated(anon):0kB isolated(file):0kB present:130314240kB mlocked:0kB dirty:1312kB writeback:0kB mapped:34904kB shmem:461336kB slab_reclaimable:2840868kB slab_unreclaimable:20564596kB kernel_stack:23512kB pagetables:63984kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jun 16 10:00:08 atlas-mds1 kernel: [515727.291547] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291548] lowmem_reserve[]:Node 1 0Normal free:83636kB min:65552kB low:81940kB high:98328kB active_anon:12781236kB inactive_anon:64544kB active_file:25411760kB inactive_file:62905916kB unevictable:23860kB isolated(anon):0kB isolated(file):0kB present:132382720kB mlocked:0kB dirty:3480kB writeback:2080kB mapped:2516kB shmem:151544kB slab_reclaimable:5494312kB slab_unreclaimable:23878592kB kernel_stack:19016kB pagetables:20128kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? no Jun 16 10:00:08 atlas-mds1 kernel: [515727.291555] 0lowmem_reserve[]: 0 0 0 0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291558] 0Node 1 0Normal free:83636kB min:65552kB low:81940kB high:98328kB active_anon:12781236kB inactive_anon:64544kB active_file:25411760kB inactive_file:62905916kB unevictable:23860kB isolated(anon):0kB isolated(file):0kB present:132382720kB mlocked:0kB dirty:3480kB writeback:2080kB mapped:2516kB shmem:151544kB slab_reclaimable:5494312kB slab_unreclaimable:23878592kB kernel_stack:19016kB pagetables:20128kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? no Jun 16 10:00:08 atlas-mds1 kernel: [515727.291567] Jun 16 10:00:08 atlas-mds1 kernel: [515727.291568] lowmem_reserve[]:Node 0 0DMA: 03*4kB 00*8kB 01*16kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291573] 1*32kB Node 0 1*64kB DMA: 0*128kB 3*4kB 1*256kB 0*8kB 0*512kB 1*16kB 1*1024kB 1*32kB 1*2048kB 1*64kB 3*4096kB 0*128kB = 15740kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291584] 1*256kB Node 0 0*512kB DMA32: 1*1024kB 10*4kB 1*2048kB 6*8kB 3*4096kB 3*16kB = 15740kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291591] 6*32kB Node 0 6*64kB DMA32: 8*128kB 10*4kB 8*256kB 6*8kB 10*512kB 3*16kB 7*1024kB 6*32kB 5*2048kB 6*64kB 88*4096kB 8*128kB = 386760kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291601] 8*256kB Node 0 10*512kB Normal: 7*1024kB 21101*4kB 5*2048kB 48*8kB 88*4096kB 2*16kB = 386760kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291608] 0*32kB Node 0 27*64kB Normal: 1*128kB 21101*4kB 0*256kB 48*8kB 1*512kB 2*16kB 1*1024kB 0*32kB 0*2048kB 27*64kB 0*4096kB 1*128kB = 88212kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291618] 0*256kB Node 1 1*512kB Normal: 1*1024kB 20020*4kB 0*2048kB 1*8kB 0*4096kB 1*16kB = 88212kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291624] 0*32kB Node 1 2*64kB Normal: 2*128kB 20020*4kB 0*256kB 1*8kB 1*512kB 1*16kB 1*1024kB 0*32kB 0*2048kB 2*64kB 0*4096kB 2*128kB = 82024kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291635] 0*256kB 41744319 total pagecache pages Jun 16 10:00:08 atlas-mds1 kernel: [515727.291637] 1*512kB 0 pages in swap cache Jun 16 10:00:08 atlas-mds1 kernel: [515727.291639] 1*1024kB Swap cache stats: add 0, delete 0, find 0/0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291641] 0*2048kB Free swap = 0kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291643] 0*4096kB Total swap = 0kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291645] = 82024kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291646] 41744319 total pagecache pages Jun 16 10:00:08 atlas-mds1 kernel: [515727.291647] 0 pages in swap cache Jun 16 10:00:08 atlas-mds1 kernel: [515727.291648] Swap cache stats: add 0, delete 0, find 0/0 Jun 16 10:00:08 atlas-mds1 kernel: [515727.291649] Free swap = 0kB Jun 16 10:00:08 atlas-mds1 kernel: [515727.291650] Total swap = 0kB Jun 16 10:00:08 atlas-mds1 kernel: [515728.164773] 67108863 pages RAM Jun 16 10:00:08 atlas-mds1 kernel: [515728.164775] 999715 pages reserved Jun 16 10:00:08 atlas-mds1 kernel: [515728.164776] 41425769 pages shared Jun 16 10:00:08 atlas-mds1 kernel: [515728.164777] 24363804 pages non-shared Jun 16 10:00:08 atlas-mds1 kernel: [515728.164870] LustreError: 12116:0:(lvfs_lib.c:151:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.4.3/lustre/lvfs/lvfs_lib.c:151 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) Jun 16 10:00:08 atlas-mds1 kernel: [515728.164875] LustreError: 12116:0:(lvfs_lib.c:151:lprocfs_stats_alloc_one()) LNET: 1500653176 total bytes allocated by lnet Jun 16 10:00:08 atlas-mds1 kernel: [515728.629106] 67108863 pages RAM Jun 16 10:00:08 atlas-mds1 kernel: [515728.629108] 999715 pages reserved Jun 16 10:00:08 atlas-mds1 kernel: [515728.629109] 41538366 pages shared Jun 16 10:00:08 atlas-mds1 kernel: [515728.629110] 24368015 pages non-shared Jun 16 10:00:08 atlas-mds1 kernel: [515728.629266] LustreError: 12124:0:(lvfs_lib.c:151:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.4.3/lustre/lvfs/lvfs_lib.c:151 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) Jun 16 10:00:08 atlas-mds1 kernel: [515728.629271] LustreError: 12124:0:(lvfs_lib.c:151:lprocfs_stats_alloc_one()) LNET: 1500684168 total bytes allocated by lnet Jun 16 10:00:08 atlas-mds1 kernel: [515729.478602] Pid: 12130, comm: ptlrpcd_18 Not tainted 2.6.32-358.23.2.el6.atlas.x86_64 #1 Jun 16 10:00:08 atlas-mds1 kernel: [515729.487953] Call Trace: Jun 16 10:00:08 atlas-mds1 kernel: [515729.490896] [] ? __alloc_pages_nodemask+0x757/0x8d0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.498405] [] ? kmem_getpages+0x62/0x170 Jun 16 10:00:08 atlas-mds1 kernel: [515729.504947] [] ? fallback_alloc+0x1ba/0x270 Jun 16 10:00:08 atlas-mds1 kernel: [515729.511668] [] ? cache_grow+0x2cf/0x320 Jun 16 10:00:08 atlas-mds1 kernel: [515729.518017] [] ? ____cache_alloc_node+0x99/0x160 Jun 16 10:00:08 atlas-mds1 kernel: [515729.525247] [] ? cfs_alloc+0x30/0x60 [libcfs] Jun 16 10:00:08 atlas-mds1 kernel: [515729.532173] [] ? __kmalloc+0x189/0x220 Jun 16 10:00:08 atlas-mds1 kernel: [515729.538426] [] ? cfs_alloc+0x30/0x60 [libcfs] Jun 16 10:00:08 atlas-mds1 kernel: [515729.545355] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] Jun 16 10:00:08 atlas-mds1 kernel: [515729.553635] [] ? lprocfs_counter_add+0x156/0x182 [lvfs] Jun 16 10:00:08 atlas-mds1 kernel: [515729.561578] [] ? after_reply+0x32a/0xd90 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515729.568895] [] ? lock_timer_base+0x3c/0x70 Jun 16 10:00:08 atlas-mds1 kernel: [515729.575569] [] ? ptlrpc_check_set+0xfd1/0x1b20 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515729.583499] [] ? ptlrpcd_check+0x53b/0x560 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515729.591041] [] ? ptlrpcd+0x233/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515729.597968] [] ? default_wake_function+0x0/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515729.605120] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515729.611850] [] ? child_rip+0xa/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515729.617845] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515729.624592] [] ? ptlrpcd+0x0/0x390 [ptlrpc] Jun 16 10:00:08 atlas-mds1 kernel: [515729.633228] [] ? child_rip+0x0/0x20 Jun 16 10:00:08 atlas-mds1 kernel: [515729.639163] Mem-Info: Jun 16 10:00:08 atlas-mds1 kernel: [515729.641892] Node 0 DMA per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515729.645617] CPU 0: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.651176] CPU 1: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.656735] CPU 2: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.662295] CPU 3: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.667848] CPU 4: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.673410] CPU 5: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.678973] CPU 6: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.684531] CPU 7: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.690091] CPU 8: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.695729] CPU 9: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.701285] CPU 10: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.706844] CPU 11: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.712401] CPU 12: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.717962] CPU 13: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.723523] CPU 14: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.729084] CPU 15: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.734642] CPU 16: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.740200] CPU 17: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.745757] CPU 18: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.751314] CPU 19: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.756872] CPU 20: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.762431] CPU 21: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.767986] CPU 22: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.773535] CPU 23: hi: 0, btch: 1 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.779092] Node 0 DMA32 per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515729.783015] CPU 0: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.788575] CPU 1: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.794133] CPU 2: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.799693] CPU 3: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.805250] CPU 4: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.810810] CPU 5: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.816366] CPU 6: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.821923] CPU 7: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.827481] CPU 8: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.833040] CPU 9: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.838594] CPU 10: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.844145] CPU 11: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.849704] CPU 12: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.855260] CPU 13: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.860823] CPU 14: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.866378] CPU 15: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.871926] CPU 16: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.877483] CPU 17: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.883042] CPU 18: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.888602] CPU 19: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.894161] CPU 20: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.899722] CPU 21: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.905280] CPU 22: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.910839] CPU 23: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.916397] Node 0 Normal per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515729.920414] CPU 0: hi: 186, btch: 31 usd: 1 Jun 16 10:00:08 atlas-mds1 kernel: [515729.925975] CPU 1: hi: 186, btch: 31 usd: 8 Jun 16 10:00:08 atlas-mds1 kernel: [515729.931536] CPU 2: hi: 186, btch: 31 usd: 67 Jun 16 10:00:08 atlas-mds1 kernel: [515729.937093] CPU 3: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.942653] CPU 4: hi: 186, btch: 31 usd: 55 Jun 16 10:00:08 atlas-mds1 kernel: [515729.948209] CPU 5: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.953769] CPU 6: hi: 186, btch: 31 usd: 1 Jun 16 10:00:08 atlas-mds1 kernel: [515729.959329] CPU 7: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.964885] CPU 8: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.970441] CPU 9: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.975997] CPU 10: hi: 186, btch: 31 usd: 36 Jun 16 10:00:08 atlas-mds1 kernel: [515729.981550] CPU 11: hi: 186, btch: 31 usd: 4 Jun 16 10:00:08 atlas-mds1 kernel: [515729.987107] CPU 12: hi: 186, btch: 31 usd: 35 Jun 16 10:00:08 atlas-mds1 kernel: [515729.992667] CPU 13: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515729.998224] CPU 14: hi: 186, btch: 31 usd: 11 Jun 16 10:00:08 atlas-mds1 kernel: [515730.003783] CPU 15: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.009336] CPU 16: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.014894] CPU 17: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.020454] CPU 18: hi: 186, btch: 31 usd: 171 Jun 16 10:00:08 atlas-mds1 kernel: [515730.026011] CPU 19: hi: 186, btch: 31 usd: 4 Jun 16 10:00:08 atlas-mds1 kernel: [515730.031574] CPU 20: hi: 186, btch: 31 usd: 82 Jun 16 10:00:08 atlas-mds1 kernel: [515730.037133] CPU 21: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.042692] CPU 22: hi: 186, btch: 31 usd: 85 Jun 16 10:00:08 atlas-mds1 kernel: [515730.048267] CPU 23: hi: 186, btch: 31 usd: 1 Jun 16 10:00:08 atlas-mds1 kernel: [515730.053824] Node 1 Normal per-cpu: Jun 16 10:00:08 atlas-mds1 kernel: [515730.057838] CPU 0: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.063398] CPU 1: hi: 186, btch: 31 usd: 5 Jun 16 10:00:08 atlas-mds1 kernel: [515730.068957] CPU 2: hi: 186, btch: 31 usd: 160 Jun 16 10:00:08 atlas-mds1 kernel: [515730.074522] CPU 3: hi: 186, btch: 31 usd: 30 Jun 16 10:00:08 atlas-mds1 kernel: [515730.080077] CPU 4: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.085627] CPU 5: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.091188] CPU 6: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.096748] CPU 7: hi: 186, btch: 31 usd: 8 Jun 16 10:00:08 atlas-mds1 kernel: [515730.102309] CPU 8: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.107867] CPU 9: hi: 186, btch: 31 usd: 58 Jun 16 10:00:08 atlas-mds1 kernel: [515730.113427] CPU 10: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.118984] CPU 11: hi: 186, btch: 31 usd: 25 Jun 16 10:00:08 atlas-mds1 kernel: [515730.124542] CPU 12: hi: 186, btch: 31 usd: 13 Jun 16 10:00:08 atlas-mds1 kernel: [515730.130099] CPU 13: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.135656] CPU 14: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.141215] CPU 15: hi: 186, btch: 31 usd: 19 Jun 16 10:00:08 atlas-mds1 kernel: [515730.146773] CPU 16: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.152329] CPU 17: hi: 186, btch: 31 usd: 30 Jun 16 10:00:08 atlas-mds1 kernel: [515730.157887] CPU 18: hi: 186, btch: 31 usd: 1 Jun 16 10:00:08 atlas-mds1 kernel: [515730.163446] CPU 19: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.169005] CPU 20: hi: 186, btch: 31 usd: 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.174564] CPU 21: hi: 186, btch: 31 usd: 55 Jun 16 10:00:08 atlas-mds1 kernel: [515730.180218] CPU 22: hi: 186, btch: 31 usd: 3 Jun 16 10:00:08 atlas-mds1 kernel: [515730.185776] CPU 23: hi: 186, btch: 31 usd: 51 Jun 16 10:00:08 atlas-mds1 kernel: [515730.191341] active_anon:10205060 inactive_anon:86396 isolated_anon:0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.191342] active_file:11640578 inactive_file:29915587 isolated_file:0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.191343] unevictable:8694 dirty:1767 writeback:144 unstable:0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.191344] free:177592 slab_reclaimable:2082589 slab_unreclaimable:11114180 Jun 16 10:00:08 atlas-mds1 kernel: [515730.191345] mapped:9355 shmem:153220 pagetables:21028 bounce:0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.228472] Node 0 DMA free:15740kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jun 16 10:00:08 atlas-mds1 kernel: [515730.269726] lowmem_reserve[]: 0 1931 129191 129191 Jun 16 10:00:08 atlas-mds1 kernel: [515730.275356] Node 0 DMA32 free:386760kB min:976kB low:1220kB high:1464kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1977504kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jun 16 10:00:08 atlas-mds1 kernel: [515730.317887] lowmem_reserve[]: 0 0 127260 127260 Jun 16 10:00:08 atlas-mds1 kernel: [515730.323218] Node 0 Normal free:221304kB min:64528kB low:80660kB high:96792kB active_anon:28039004kB inactive_anon:281040kB active_file:21150536kB inactive_file:56790500kB unevictable:10916kB isolated(anon):0kB isolated(file):0kB present:130314240kB mlocked:0kB dirty:3632kB writeback:0kB mapped:34904kB shmem:461336kB slab_reclaimable:2837424kB slab_unreclaimable:20540152kB kernel_stack:23520kB pagetables:63984kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jun 16 10:00:08 atlas-mds1 kernel: [515730.372548] lowmem_reserve[]: 0 0 0 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.376905] Node 1 Normal free:81604kB min:65552kB low:81940kB high:98328kB active_anon:12781236kB inactive_anon:64544kB active_file:25411776kB inactive_file:62875844kB unevictable:23860kB isolated(anon):0kB isolated(file):0kB present:132382720kB mlocked:0kB dirty:4100kB writeback:576kB mapped:2516kB shmem:151544kB slab_reclaimable:5492932kB slab_unreclaimable:23917012kB kernel_stack:19016kB pagetables:20128kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jun 16 10:00:08 atlas-mds1 kernel: [515730.426135] lowmem_reserve[]: 0 0 0 0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.430498] Node 0 DMA: 3*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB Jun 16 10:00:08 atlas-mds1 kernel: [515730.442864] Node 0 DMA32: 10*4kB 6*8kB 3*16kB 6*32kB 6*64kB 8*128kB 8*256kB 10*512kB 7*1024kB 5*2048kB 88*4096kB = 386760kB Jun 16 10:00:08 atlas-mds1 kernel: [515730.455812] Node 0 Normal: 21833*4kB 333*8kB 169*16kB 111*32kB 258*64kB 1*128kB 0*256kB 8*512kB 90*1024kB 0*2048kB 0*4096kB = 209148kB Jun 16 10:00:08 atlas-mds1 kernel: [515730.469824] Node 1 Normal: 18156*4kB 58*8kB 1*16kB 2*32kB 3*64kB 4*128kB 1*256kB 13*512kB 1*1024kB 0*2048kB 0*4096kB = 81808kB Jun 16 10:00:08 atlas-mds1 kernel: [515730.483064] 41714127 total pagecache pages Jun 16 10:00:08 atlas-mds1 kernel: [515730.487851] 0 pages in swap cache Jun 16 10:00:08 atlas-mds1 kernel: [515730.491749] Swap cache stats: add 0, delete 0, find 0/0 Jun 16 10:00:08 atlas-mds1 kernel: [515730.497793] Free swap = 0kB Jun 16 10:00:08 atlas-mds1 kernel: [515730.501213] Total swap = 0kB Jun 16 10:00:08 atlas-mds1 kernel: [515731.376953] 67108863 pages RAM Jun 16 10:00:08 atlas-mds1 kernel: [515731.380570] 999715 pages reserved Jun 16 10:00:08 atlas-mds1 kernel: [515731.384474] 42168207 pages shared Jun 16 10:00:08 atlas-mds1 kernel: [515731.388372] 24373437 pages non-shared Jun 16 10:00:08 atlas-mds1 kernel: [515731.392830] LustreError: 12130:0:(lvfs_lib.c:151:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.4.3/lustre/lvfs/lvfs_lib.c:151 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) Jun 16 10:00:08 atlas-mds1 kernel: [515731.416965] LustreError: 12130:0:(lvfs_lib.c:151:lprocfs_stats_alloc_one()) LNET: 1500663096 total bytes allocated by lnet