crash: /var/crash/2014-01-14-01:30/vmlinux-3.0.51-0.7.9-default.gz: original filename unknown Use "-f /var/crash/2014-01-14-01:30/vmlinux-3.0.51-0.7.9-default.gz" on command line to prevent this message. " ================== " " CRASH SUMMARY " "================== " KERNEL: /var/crash/2014-01-14-01:30/vmlinux-3.0.51-0.7.9-default.gz DEBUGINFO: /var/crash/2014-01-14-01:30/vmlinux-3.0.51-0.7.9-default.debug DUMPFILE: /var/crash/2014-01-14-01:30/vmcore [PARTIAL DUMP] CPUS: 32 DATE: Tue Jan 14 01:30:02 2014 UPTIME: 4 days, 03:26:41 LOAD AVERAGE: 0.01, 0.05, 0.05 TASKS: 689 NODENAME: container-nas4 RELEASE: 3.0.51-0.7.9-default VERSION: #1 SMP Thu Nov 29 22:12:17 UTC 2012 (f3be9d0) MACHINE: x86_64 (2194 Mhz) MEMORY: 64 GB PANIC: "[357548.960705] Kernel panic - not syncing: LBUG" " ================== " " MEMORY USAGE " "================== " PAGES TOTAL PERCENTAGE TOTAL MEM 16383233 62.5 GB ---- FREE 110384 431.2 MB 0% of TOTAL MEM USED 16272849 62.1 GB 99% of TOTAL MEM SHARED 6770194 25.8 GB 41% of TOTAL MEM BUFFERS 40 160 KB 0% of TOTAL MEM CACHED 15301053 58.4 GB 93% of TOTAL MEM SLAB 446575 1.7 GB 2% of TOTAL MEM TOTAL SWAP 2097151 8 GB ---- SWAP USED 18 72 KB 0% of TOTAL SWAP SWAP FREE 2097133 8 GB 99% of TOTAL SWAP " ================== " " CONSOLE LOG " "================== " 186, btch: 31 usd: 3 [36142.259338] CPU 2: hi: 186, btch: 31 usd: 181 [36142.259340] CPU 3: hi: 186, btch: 31 usd: 175 [36142.259343] CPU 4: hi: 186, btch: 31 usd: 169 [36142.259345] CPU 5: hi: 186, btch: 31 usd: 172 [36142.259348] CPU 6: hi: 186, btch: 31 usd: 156 [36142.259350] CPU 7: hi: 186, btch: 31 usd: 167 [36142.259352] CPU 8: hi: 186, btch: 31 usd: 154 [36142.259355] CPU 9: hi: 186, btch: 31 usd: 89 [36142.259357] CPU 10: hi: 186, btch: 31 usd: 156 [36142.259360] CPU 11: hi: 186, btch: 31 usd: 178 [36142.259362] CPU 12: hi: 186, btch: 31 usd: 169 [36142.259364] CPU 13: hi: 186, btch: 31 usd: 176 [36142.259366] CPU 14: hi: 186, btch: 31 usd: 155 [36142.259367] CPU 15: hi: 186, btch: 31 usd: 168 [36142.259369] CPU 16: hi: 186, btch: 31 usd: 162 [36142.259370] CPU 17: hi: 186, btch: 31 usd: 185 [36142.259372] CPU 18: hi: 186, btch: 31 usd: 158 [36142.259374] CPU 19: hi: 186, btch: 31 usd: 175 [36142.259375] CPU 20: hi: 186, btch: 31 usd: 168 [36142.259377] CPU 21: hi: 186, btch: 31 usd: 185 [36142.259378] CPU 22: hi: 186, btch: 31 usd: 79 [36142.259380] CPU 23: hi: 186, btch: 31 usd: 178 [36142.259382] CPU 24: hi: 186, btch: 31 usd: 96 [36142.259383] CPU 25: hi: 186, btch: 31 usd: 163 [36142.259385] CPU 26: hi: 186, btch: 31 usd: 175 [36142.259386] CPU 27: hi: 186, btch: 31 usd: 182 [36142.259388] CPU 28: hi: 186, btch: 31 usd: 171 [36142.259389] CPU 29: hi: 186, btch: 31 usd: 171 [36142.259391] CPU 30: hi: 186, btch: 31 usd: 164 [36142.259392] CPU 31: hi: 186, btch: 31 usd: 140 [36142.259394] Node 1 Normal per-cpu: [36142.259396] CPU 0: hi: 186, btch: 31 usd: 0 [36142.259397] CPU 1: hi: 186, btch: 31 usd: 30 [36142.259400] CPU 2: hi: 186, btch: 31 usd: 177 [36142.259402] CPU 3: hi: 186, btch: 31 usd: 158 [36142.259405] CPU 4: hi: 186, btch: 31 usd: 182 [36142.259407] CPU 5: hi: 186, btch: 31 usd: 57 [36142.259410] CPU 6: hi: 186, btch: 31 usd: 180 [36142.259412] CPU 7: hi: 186, btch: 31 usd: 162 [36142.259415] CPU 8: hi: 186, btch: 31 usd: 170 [36142.259417] CPU 9: hi: 186, btch: 31 usd: 180 [36142.259420] CPU 10: hi: 186, btch: 31 usd: 179 [36142.259422] CPU 11: hi: 186, btch: 31 usd: 162 [36142.259425] CPU 12: hi: 186, btch: 31 usd: 160 [36142.259427] CPU 13: hi: 186, btch: 31 usd: 155 [36142.259429] CPU 14: hi: 186, btch: 31 usd: 164 [36142.259431] CPU 15: hi: 186, btch: 31 usd: 173 [36142.259432] CPU 16: hi: 186, btch: 31 usd: 172 [36142.259434] CPU 17: hi: 186, btch: 31 usd: 178 [36142.259436] CPU 18: hi: 186, btch: 31 usd: 35 [36142.259437] CPU 19: hi: 186, btch: 31 usd: 41 [36142.259439] CPU 20: hi: 186, btch: 31 usd: 179 [36142.259440] CPU 21: hi: 186, btch: 31 usd: 174 [36142.259442] CPU 22: hi: 186, btch: 31 usd: 182 [36142.259444] CPU 23: hi: 186, btch: 31 usd: 179 [36142.259446] CPU 24: hi: 186, btch: 31 usd: 179 [36142.259447] CPU 25: hi: 186, btch: 31 usd: 155 [36142.259449] CPU 26: hi: 186, btch: 31 usd: 163 [36142.259451] CPU 27: hi: 186, btch: 31 usd: 185 [36142.259452] CPU 28: hi: 186, btch: 31 usd: 156 [36142.259454] CPU 29: hi: 186, btch: 31 usd: 157 [36142.259456] CPU 30: hi: 186, btch: 31 usd: 166 [36142.259457] CPU 31: hi: 186, btch: 31 usd: 165 [36142.259462] active_anon:164154 inactive_anon:26136 isolated_anon:0 [36142.259463] active_file:16253 inactive_file:15221922 isolated_file:32 [36142.259464] unevictable:2648 dirty:1029 writeback:301 unstable:96059 [36142.259465] free:78010 slab_reclaimable:48886 slab_unreclaimable:467616 [36142.259466] mapped:12046 shmem:301 pagetables:2475 bounce:0 [36142.259467] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [36142.259476] lowmem_reserve[]: 0 2959 32249 32249 [36142.259479] Node 0 DMA32 free:180260kB min:4128kB low:5160kB high:6192kB active_anon:40kB inactive_anon:40964kB active_file:348kB inactive_file:1793576kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:436kB writeback:1744kB mapped:0kB shmem:0kB slab_reclaimable:40736kB slab_unreclaimable:404372kB kernel_stack:40kB pagetables:0kB unstable:12184kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [36142.259489] lowmem_reserve[]: 0 0 29290 29290 [36142.259491] Node 0 Normal free:55180kB min:40864kB low:51080kB high:61296kB active_anon:461056kB inactive_anon:32660kB active_file:37960kB inactive_file:27306232kB unevictable:8528kB isolated(anon):0kB isolated(file):128kB present:29992960kB mlocked:2396kB dirty:2444kB writeback:0kB mapped:30388kB shmem:300kB slab_reclaimable:108944kB slab_unreclaimable:989740kB kernel_stack:2288kB pagetables:4964kB unstable:250000kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [36142.259501] lowmem_reserve[]: 0 0 0 0 [36142.259504] Node 1 Normal free:61128kB min:45092kB low:56364kB high:67636kB active_anon:195520kB inactive_anon:30920kB active_file:26704kB inactive_file:31787112kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:1236kB writeback:60kB mapped:17796kB shmem:904kB slab_reclaimable:45864kB slab_unreclaimable:476352kB kernel_stack:3016kB pagetables:4936kB unstable:122052kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [36142.259518] lowmem_reserve[]: 0 0 0 0 [36142.259522] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [36142.259531] Node 0 DMA32: 45212*4kB 30*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 181104kB [36142.259538] Node 0 Normal: 13590*4kB 77*8kB 48*16kB 19*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56544kB [36142.259545] Node 1 Normal: 14375*4kB 95*8kB 145*16kB 66*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62692kB [36142.259553] 15238538 total pagecache pages [36142.259554] 0 pages in swap cache [36142.259556] Swap cache stats: add 0, delete 0, find 0/0 [36142.259557] Free swap = 8388604kB [36142.259558] Total swap = 8388604kB [36142.555904] 16777200 pages RAM [36142.555909] 393967 pages reserved [36142.555912] 1632243 pages shared [36142.555914] 14749348 pages non-shared [36142.555976] LustreError: 9216:0:(osc_request.c:2198:osc_build_rpc()) prep_req failed: -12 [36142.556346] LustreError: 9216:0:(osc_cache.c:2211:osc_check_rpcs()) Write request failed with -12 [36392.575091] The following is only an harmless informational message. [36392.575099] Unless you get a _continuous_flood_ of these messages it means [36392.575102] everything is working fine. Allocations from irqs cannot be [36392.575104] perfectly reliable and the kernel is designed to handle that. [36392.575110] ptlrpcd_13: page allocation failure: order:1, mode:0x40 [36392.575116] Pid: 9228, comm: ptlrpcd_13 Tainted: P NX 3.0.51-0.7.9-default #1 [36392.575120] Call Trace: [36392.575149] [] dump_trace+0x75/0x300 [36392.575162] [] dump_stack+0x69/0x6f [36392.575174] [] warn_alloc_failed+0x102/0x1a0 [36392.575183] [] __alloc_pages_slowpath+0x541/0x7f0 [36392.575191] [] __alloc_pages_nodemask+0x1e9/0x200 [36392.575201] [] kmem_getpages+0x56/0x170 [36392.575209] [] fallback_alloc+0x19b/0x270 [36392.575216] [] __kmalloc+0x285/0x330 [36392.575296] [] ptlrpc_new_bulk+0x45/0x2a0 [ptlrpc] [36392.575399] [] ptlrpc_prep_bulk_imp+0x62/0x1b0 [ptlrpc] [36392.575446] [] osc_brw_prep_request+0x1bf/0xf20 [osc] [36392.575472] [] osc_build_rpc+0x7ba/0x1380 [osc] [36392.575501] [] osc_send_write_rpc+0x1e5/0x470 [osc] [36392.575543] [] osc_check_rpcs+0x5b4/0x710 [osc] [36392.575589] [] osc_io_unplug0+0x35b/0x4c0 [osc] [36392.575609] [] brw_queue_work+0x30/0xd0 [osc] [36392.575656] [] ptlrpc_check_set+0x522/0x1ae0 [ptlrpc] [36392.575741] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [36392.575843] [] ptlrpcd+0x32b/0x410 [ptlrpc] [36392.575902] [] kthread+0x96/0xa0 [36392.575912] [] kernel_thread_helper+0x4/0x10 [36392.575919] Mem-Info: [36392.575921] Node 0 DMA per-cpu: [36392.575926] CPU 0: hi: 0, btch: 1 usd: 0 [36392.575929] CPU 1: hi: 0, btch: 1 usd: 0 [36392.575932] CPU 2: hi: 0, btch: 1 usd: 0 [36392.575935] CPU 3: hi: 0, btch: 1 usd: 0 [36392.575938] CPU 4: hi: 0, btch: 1 usd: 0 [36392.575941] CPU 5: hi: 0, btch: 1 usd: 0 [36392.575944] CPU 6: hi: 0, btch: 1 usd: 0 [36392.575947] CPU 7: hi: 0, btch: 1 usd: 0 [36392.575950] CPU 8: hi: 0, btch: 1 usd: 0 [36392.575952] CPU 9: hi: 0, btch: 1 usd: 0 [36392.575956] CPU 10: hi: 0, btch: 1 usd: 0 [36392.575958] CPU 11: hi: 0, btch: 1 usd: 0 [36392.575961] CPU 12: hi: 0, btch: 1 usd: 0 [36392.575964] CPU 13: hi: 0, btch: 1 usd: 0 [36392.575967] CPU 14: hi: 0, btch: 1 usd: 0 [36392.575970] CPU 15: hi: 0, btch: 1 usd: 0 [36392.575973] CPU 16: hi: 0, btch: 1 usd: 0 [36392.575976] CPU 17: hi: 0, btch: 1 usd: 0 [36392.575979] CPU 18: hi: 0, btch: 1 usd: 0 [36392.575982] CPU 19: hi: 0, btch: 1 usd: 0 [36392.575985] CPU 20: hi: 0, btch: 1 usd: 0 [36392.575988] CPU 21: hi: 0, btch: 1 usd: 0 [36392.575991] CPU 22: hi: 0, btch: 1 usd: 0 [36392.575993] CPU 23: hi: 0, btch: 1 usd: 0 [36392.575996] CPU 24: hi: 0, btch: 1 usd: 0 [36392.575999] CPU 25: hi: 0, btch: 1 usd: 0 [36392.576002] CPU 26: hi: 0, btch: 1 usd: 0 [36392.576005] CPU 27: hi: 0, btch: 1 usd: 0 [36392.576008] CPU 28: hi: 0, btch: 1 usd: 0 [36392.576011] CPU 29: hi: 0, btch: 1 usd: 0 [36392.576014] CPU 30: hi: 0, btch: 1 usd: 0 [36392.576017] CPU 31: hi: 0, btch: 1 usd: 0 [36392.576019] Node 0 DMA32 per-cpu: [36392.576023] CPU 0: hi: 186, btch: 31 usd: 165 [36392.576026] CPU 1: hi: 186, btch: 31 usd: 177 [36392.576029] CPU 2: hi: 186, btch: 31 usd: 172 [36392.576032] CPU 3: hi: 186, btch: 31 usd: 161 [36392.576035] CPU 4: hi: 186, btch: 31 usd: 12 [36392.576038] CPU 5: hi: 186, btch: 31 usd: 47 [36392.576041] CPU 6: hi: 186, btch: 31 usd: 46 [36392.576045] CPU 7: hi: 186, btch: 31 usd: 172 [36392.576047] CPU 8: hi: 186, btch: 31 usd: 108 [36392.576050] CPU 9: hi: 186, btch: 31 usd: 143 [36392.576053] CPU 10: hi: 186, btch: 31 usd: 156 [36392.576056] CPU 11: hi: 186, btch: 31 usd: 43 [36392.576059] CPU 12: hi: 186, btch: 31 usd: 138 [36392.576062] CPU 13: hi: 186, btch: 31 usd: 166 [36392.576065] CPU 14: hi: 186, btch: 31 usd: 43 [36392.576068] CPU 15: hi: 186, btch: 31 usd: 166 [36392.576071] CPU 16: hi: 186, btch: 31 usd: 165 [36392.576074] CPU 17: hi: 186, btch: 31 usd: 156 [36392.576077] CPU 18: hi: 186, btch: 31 usd: 160 [36392.576080] CPU 19: hi: 186, btch: 31 usd: 169 [36392.576083] CPU 20: hi: 186, btch: 31 usd: 157 [36392.576086] CPU 21: hi: 186, btch: 31 usd: 160 [36392.576089] CPU 22: hi: 186, btch: 31 usd: 180 [36392.576092] CPU 23: hi: 186, btch: 31 usd: 184 [36392.576095] CPU 24: hi: 186, btch: 31 usd: 0 [36392.576098] CPU 25: hi: 186, btch: 31 usd: 116 [36392.576101] CPU 26: hi: 186, btch: 31 usd: 92 [36392.576104] CPU 27: hi: 186, btch: 31 usd: 87 [36392.576106] CPU 28: hi: 186, btch: 31 usd: 0 [36392.576109] CPU 29: hi: 186, btch: 31 usd: 74 [36392.576112] CPU 30: hi: 186, btch: 31 usd: 171 [36392.576115] CPU 31: hi: 186, btch: 31 usd: 58 [36392.576118] Node 0 Normal per-cpu: [36392.576121] CPU 0: hi: 186, btch: 31 usd: 156 [36392.576124] CPU 1: hi: 186, btch: 31 usd: 186 [36392.576127] CPU 2: hi: 186, btch: 31 usd: 184 [36392.576130] CPU 3: hi: 186, btch: 31 usd: 145 [36392.576132] CPU 4: hi: 186, btch: 31 usd: 38 [36392.576135] CPU 5: hi: 186, btch: 31 usd: 52 [36392.576138] CPU 6: hi: 186, btch: 31 usd: 156 [36392.576141] CPU 7: hi: 186, btch: 31 usd: 161 [36392.576144] CPU 8: hi: 186, btch: 31 usd: 142 [36392.576147] CPU 9: hi: 186, btch: 31 usd: 164 [36392.576150] CPU 10: hi: 186, btch: 31 usd: 207 [36392.576153] CPU 11: hi: 186, btch: 31 usd: 99 [36392.576156] CPU 12: hi: 186, btch: 31 usd: 160 [36392.576159] CPU 13: hi: 186, btch: 31 usd: 160 [36392.576161] CPU 14: hi: 186, btch: 31 usd: 159 [36392.576165] CPU 15: hi: 186, btch: 31 usd: 169 [36392.576167] CPU 16: hi: 186, btch: 31 usd: 157 [36392.576170] CPU 17: hi: 186, btch: 31 usd: 181 [36392.576173] CPU 18: hi: 186, btch: 31 usd: 167 [36392.576176] CPU 19: hi: 186, btch: 31 usd: 161 [36392.576179] CPU 20: hi: 186, btch: 31 usd: 174 [36392.576182] CPU 21: hi: 186, btch: 31 usd: 155 [36392.576185] CPU 22: hi: 186, btch: 31 usd: 168 [36392.576188] CPU 23: hi: 186, btch: 31 usd: 156 [36392.576190] CPU 24: hi: 186, btch: 31 usd: 0 [36392.576193] CPU 25: hi: 186, btch: 31 usd: 35 [36392.576196] CPU 26: hi: 186, btch: 31 usd: 51 [36392.576199] CPU 27: hi: 186, btch: 31 usd: 175 [36392.576202] CPU 28: hi: 186, btch: 31 usd: 1 [36392.576205] CPU 29: hi: 186, btch: 31 usd: 182 [36392.576208] CPU 30: hi: 186, btch: 31 usd: 159 [36392.576211] CPU 31: hi: 186, btch: 31 usd: 162 [36392.576213] Node 1 Normal per-cpu: [36392.576217] CPU 0: hi: 186, btch: 31 usd: 154 [36392.576220] CPU 1: hi: 186, btch: 31 usd: 168 [36392.576223] CPU 2: hi: 186, btch: 31 usd: 166 [36392.576226] CPU 3: hi: 186, btch: 31 usd: 163 [36392.576229] CPU 4: hi: 186, btch: 31 usd: 30 [36392.576232] CPU 5: hi: 186, btch: 31 usd: 25 [36392.576236] CPU 6: hi: 186, btch: 31 usd: 0 [36392.576240] CPU 7: hi: 186, btch: 31 usd: 173 [36392.576244] CPU 8: hi: 186, btch: 31 usd: 164 [36392.576248] CPU 9: hi: 186, btch: 31 usd: 175 [36392.576252] CPU 10: hi: 186, btch: 31 usd: 183 [36392.576256] CPU 11: hi: 186, btch: 31 usd: 169 [36392.576260] CPU 12: hi: 186, btch: 31 usd: 184 [36392.576264] CPU 13: hi: 186, btch: 31 usd: 162 [36392.576268] CPU 14: hi: 186, btch: 31 usd: 177 [36392.576272] CPU 15: hi: 186, btch: 31 usd: 174 [36392.576276] CPU 16: hi: 186, btch: 31 usd: 0 [36392.576280] CPU 17: hi: 186, btch: 31 usd: 125 [36392.576283] CPU 18: hi: 186, btch: 31 usd: 163 [36392.576287] CPU 19: hi: 186, btch: 31 usd: 171 [36392.576291] CPU 20: hi: 186, btch: 31 usd: 37 [36392.576295] CPU 21: hi: 186, btch: 31 usd: 166 [36392.576299] CPU 22: hi: 186, btch: 31 usd: 110 [36392.576303] CPU 23: hi: 186, btch: 31 usd: 164 [36392.576307] CPU 24: hi: 186, btch: 31 usd: 174 [36392.576311] CPU 25: hi: 186, btch: 31 usd: 158 [36392.576315] CPU 26: hi: 186, btch: 31 usd: 72 [36392.576319] CPU 27: hi: 186, btch: 31 usd: 168 [36392.576323] CPU 28: hi: 186, btch: 31 usd: 35 [36392.576327] CPU 29: hi: 186, btch: 31 usd: 168 [36392.576331] CPU 30: hi: 186, btch: 31 usd: 162 [36392.576334] CPU 31: hi: 186, btch: 31 usd: 180 [36392.576345] active_anon:164406 inactive_anon:26136 isolated_anon:0 [36392.576347] active_file:15957 inactive_file:15248062 isolated_file:32 [36392.576349] unevictable:2648 dirty:981 writeback:1211 unstable:73264 [36392.576351] free:68647 slab_reclaimable:48444 slab_unreclaimable:452078 [36392.576353] mapped:12069 shmem:301 pagetables:2531 bounce:0 [36392.576358] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [36392.576377] lowmem_reserve[]: 0 2959 32249 32249 [36392.576384] Node 0 DMA32 free:123364kB min:4128kB low:5160kB high:6192kB active_anon:48kB inactive_anon:40964kB active_file:24kB inactive_file:1804376kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:544kB writeback:1156kB mapped:0kB shmem:0kB slab_reclaimable:40020kB slab_unreclaimable:453672kB kernel_stack:40kB pagetables:0kB unstable:11176kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [36392.576405] lowmem_reserve[]: 0 0 29290 29290 [36392.576411] Node 0 Normal free:59656kB min:40864kB low:51080kB high:61296kB active_anon:461068kB inactive_anon:32660kB active_file:36552kB inactive_file:27102884kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:2072kB writeback:2296kB mapped:30476kB shmem:300kB slab_reclaimable:108184kB slab_unreclaimable:1192880kB kernel_stack:2304kB pagetables:4964kB unstable:216576kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [36392.576432] lowmem_reserve[]: 0 0 0 0 [36392.576438] Node 1 Normal free:76096kB min:45092kB low:56364kB high:67636kB active_anon:196508kB inactive_anon:30920kB active_file:27252kB inactive_file:32084604kB unevictable:2064kB isolated(anon):0kB isolated(file):128kB present:33095680kB mlocked:2064kB dirty:1308kB writeback:1236kB mapped:17800kB shmem:904kB slab_reclaimable:45572kB slab_unreclaimable:161760kB kernel_stack:3024kB pagetables:5160kB unstable:65304kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [36392.576459] lowmem_reserve[]: 0 0 0 0 [36392.576465] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [36392.576481] Node 0 DMA32: 30656*4kB 72*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 123232kB [36392.576498] Node 0 Normal: 14274*4kB 328*8kB 16*16kB 21*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60648kB [36392.576515] Node 1 Normal: 14091*4kB 1992*8kB 245*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76284kB [36392.576531] 15264928 total pagecache pages [36392.576535] 0 pages in swap cache [36392.576538] Swap cache stats: add 0, delete 0, find 0/0 [36392.576541] Free swap = 8388604kB [36392.576544] Total swap = 8388604kB [36392.861262] 16777200 pages RAM [36392.861266] 393967 pages reserved [36392.861268] 1609961 pages shared [36392.861270] 14762975 pages non-shared [36392.861333] LustreError: 9228:0:(osc_request.c:2198:osc_build_rpc()) prep_req failed: -12 [36392.861820] LustreError: 9228:0:(osc_cache.c:2211:osc_check_rpcs()) Write request failed with -12 [36531.064420] The following is only an harmless informational message. [36531.064428] Unless you get a _continuous_flood_ of these messages it means [36531.064431] everything is working fine. Allocations from irqs cannot be [36531.064434] perfectly reliable and the kernel is designed to handle that. [36531.064439] ptlrpcd_31: page allocation failure: order:1, mode:0x40 [36531.064445] Pid: 9246, comm: ptlrpcd_31 Tainted: P NX 3.0.51-0.7.9-default #1 [36531.064449] Call Trace: [36531.064486] [] dump_trace+0x75/0x300 [36531.064499] [] dump_stack+0x69/0x6f [36531.064510] [] warn_alloc_failed+0x102/0x1a0 [36531.064518] [] __alloc_pages_slowpath+0x541/0x7f0 [36531.064527] [] __alloc_pages_nodemask+0x1e9/0x200 [36531.064536] [] kmem_getpages+0x56/0x170 [36531.064544] [] fallback_alloc+0x19b/0x270 [36531.064551] [] __kmalloc+0x285/0x330 [36531.064580] [] LNetMDAttach+0x163/0x5b0 [lnet] [36531.064689] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [36531.064769] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [36531.064847] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [36531.064920] [] ptlrpc_check_set+0x408/0x1ae0 [ptlrpc] [36531.065004] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [36531.065106] [] ptlrpcd+0x27b/0x410 [ptlrpc] [36531.065165] [] kthread+0x96/0xa0 [36531.065175] [] kernel_thread_helper+0x4/0x10 [36531.065181] Mem-Info: [36531.065183] Node 0 DMA per-cpu: [36531.065188] CPU 0: hi: 0, btch: 1 usd: 0 [36531.065191] CPU 1: hi: 0, btch: 1 usd: 0 [36531.065194] CPU 2: hi: 0, btch: 1 usd: 0 [36531.065197] CPU 3: hi: 0, btch: 1 usd: 0 [36531.065200] CPU 4: hi: 0, btch: 1 usd: 0 [36531.065203] CPU 5: hi: 0, btch: 1 usd: 0 [36531.065206] CPU 6: hi: 0, btch: 1 usd: 0 [36531.065208] CPU 7: hi: 0, btch: 1 usd: 0 [36531.065211] CPU 8: hi: 0, btch: 1 usd: 0 [36531.065214] CPU 9: hi: 0, btch: 1 usd: 0 [36531.065217] CPU 10: hi: 0, btch: 1 usd: 0 [36531.065220] CPU 11: hi: 0, btch: 1 usd: 0 [36531.065223] CPU 12: hi: 0, btch: 1 usd: 0 [36531.065226] CPU 13: hi: 0, btch: 1 usd: 0 [36531.065229] CPU 14: hi: 0, btch: 1 usd: 0 [36531.065232] CPU 15: hi: 0, btch: 1 usd: 0 [36531.065234] CPU 16: hi: 0, btch: 1 usd: 0 [36531.065237] CPU 17: hi: 0, btch: 1 usd: 0 [36531.065240] CPU 18: hi: 0, btch: 1 usd: 0 [36531.065243] CPU 19: hi: 0, btch: 1 usd: 0 [36531.065246] CPU 20: hi: 0, btch: 1 usd: 0 [36531.065249] CPU 21: hi: 0, btch: 1 usd: 0 [36531.065252] CPU 22: hi: 0, btch: 1 usd: 0 [36531.065255] CPU 23: hi: 0, btch: 1 usd: 0 [36531.065258] CPU 24: hi: 0, btch: 1 usd: 0 [36531.065260] CPU 25: hi: 0, btch: 1 usd: 0 [36531.065263] CPU 26: hi: 0, btch: 1 usd: 0 [36531.065266] CPU 27: hi: 0, btch: 1 usd: 0 [36531.065269] CPU 28: hi: 0, btch: 1 usd: 0 [36531.065272] CPU 29: hi: 0, btch: 1 usd: 0 [36531.065275] CPU 30: hi: 0, btch: 1 usd: 0 [36531.065278] CPU 31: hi: 0, btch: 1 usd: 0 [36531.065280] Node 0 DMA32 per-cpu: [36531.065284] CPU 0: hi: 186, btch: 31 usd: 190 [36531.065287] CPU 1: hi: 186, btch: 31 usd: 44 [36531.065290] CPU 2: hi: 186, btch: 31 usd: 158 [36531.065293] CPU 3: hi: 186, btch: 31 usd: 177 [36531.065296] CPU 4: hi: 186, btch: 31 usd: 155 [36531.065299] CPU 5: hi: 186, btch: 31 usd: 178 [36531.065302] CPU 6: hi: 186, btch: 31 usd: 173 [36531.065305] CPU 7: hi: 186, btch: 31 usd: 173 [36531.065308] CPU 8: hi: 186, btch: 31 usd: 39 [36531.065311] CPU 9: hi: 186, btch: 31 usd: 4 [36531.065314] CPU 10: hi: 186, btch: 31 usd: 131 [36531.065317] CPU 11: hi: 186, btch: 31 usd: 146 [36531.065319] CPU 12: hi: 186, btch: 31 usd: 166 [36531.065322] CPU 13: hi: 186, btch: 31 usd: 181 [36531.065325] CPU 14: hi: 186, btch: 31 usd: 180 [36531.065328] CPU 15: hi: 186, btch: 31 usd: 137 [36531.065331] CPU 16: hi: 186, btch: 31 usd: 171 [36531.065334] CPU 17: hi: 186, btch: 31 usd: 173 [36531.065337] CPU 18: hi: 186, btch: 31 usd: 153 [36531.065340] CPU 19: hi: 186, btch: 31 usd: 163 [36531.065343] CPU 20: hi: 186, btch: 31 usd: 158 [36531.065346] CPU 21: hi: 186, btch: 31 usd: 169 [36531.065349] CPU 22: hi: 186, btch: 31 usd: 155 [36531.065352] CPU 23: hi: 186, btch: 31 usd: 174 [36531.065355] CPU 24: hi: 186, btch: 31 usd: 151 [36531.065358] CPU 25: hi: 186, btch: 31 usd: 78 [36531.065361] CPU 26: hi: 186, btch: 31 usd: 150 [36531.065364] CPU 27: hi: 186, btch: 31 usd: 57 [36531.065366] CPU 28: hi: 186, btch: 31 usd: 171 [36531.065370] CPU 29: hi: 186, btch: 31 usd: 17 [36531.065372] CPU 30: hi: 186, btch: 31 usd: 158 [36531.065375] CPU 31: hi: 186, btch: 31 usd: 81 [36531.065378] Node 0 Normal per-cpu: [36531.065381] CPU 0: hi: 186, btch: 31 usd: 178 [36531.065384] CPU 1: hi: 186, btch: 31 usd: 17 [36531.065387] CPU 2: hi: 186, btch: 31 usd: 181 [36531.065390] CPU 3: hi: 186, btch: 31 usd: 169 [36531.065393] CPU 4: hi: 186, btch: 31 usd: 174 [36531.065395] CPU 5: hi: 186, btch: 31 usd: 166 [36531.065398] CPU 6: hi: 186, btch: 31 usd: 173 [36531.065401] CPU 7: hi: 186, btch: 31 usd: 160 [36531.065404] CPU 8: hi: 186, btch: 31 usd: 0 [36531.065407] CPU 9: hi: 186, btch: 31 usd: 178 [36531.065410] CPU 10: hi: 186, btch: 31 usd: 164 [36531.065412] CPU 11: hi: 186, btch: 31 usd: 161 [36531.065415] CPU 12: hi: 186, btch: 31 usd: 171 [36531.065418] CPU 13: hi: 186, btch: 31 usd: 181 [36531.065421] CPU 14: hi: 186, btch: 31 usd: 156 [36531.065424] CPU 15: hi: 186, btch: 31 usd: 183 [36531.065427] CPU 16: hi: 186, btch: 31 usd: 186 [36531.065430] CPU 17: hi: 186, btch: 31 usd: 186 [36531.065433] CPU 18: hi: 186, btch: 31 usd: 111 [36531.065436] CPU 19: hi: 186, btch: 31 usd: 169 [36531.065439] CPU 20: hi: 186, btch: 31 usd: 156 [36531.065442] CPU 21: hi: 186, btch: 31 usd: 156 [36531.065444] CPU 22: hi: 186, btch: 31 usd: 160 [36531.065447] CPU 23: hi: 186, btch: 31 usd: 167 [36531.065450] CPU 24: hi: 186, btch: 31 usd: 143 [36531.065453] CPU 25: hi: 186, btch: 31 usd: 183 [36531.065456] CPU 26: hi: 186, btch: 31 usd: 151 [36531.065459] CPU 27: hi: 186, btch: 31 usd: 117 [36531.065461] CPU 28: hi: 186, btch: 31 usd: 184 [36531.065464] CPU 29: hi: 186, btch: 31 usd: 158 [36531.065468] CPU 30: hi: 186, btch: 31 usd: 181 [36531.065470] CPU 31: hi: 186, btch: 31 usd: 92 [36531.065473] Node 1 Normal per-cpu: [36531.065477] CPU 0: hi: 186, btch: 31 usd: 51 [36531.065480] CPU 1: hi: 186, btch: 31 usd: 48 [36531.065483] CPU 2: hi: 186, btch: 31 usd: 160 [36531.065486] CPU 3: hi: 186, btch: 31 usd: 127 [36531.065489] CPU 4: hi: 186, btch: 31 usd: 184 [36531.065492] CPU 5: hi: 186, btch: 31 usd: 183 [36531.065495] CPU 6: hi: 186, btch: 31 usd: 165 [36531.065498] CPU 7: hi: 186, btch: 31 usd: 181 [36531.065500] CPU 8: hi: 186, btch: 31 usd: 30 [36531.065503] CPU 9: hi: 186, btch: 31 usd: 18 [36531.065506] CPU 10: hi: 186, btch: 31 usd: 174 [36531.065509] CPU 11: hi: 186, btch: 31 usd: 161 [36531.065512] CPU 12: hi: 186, btch: 31 usd: 159 [36531.065515] CPU 13: hi: 186, btch: 31 usd: 167 [36531.065518] CPU 14: hi: 186, btch: 31 usd: 173 [36531.065521] CPU 15: hi: 186, btch: 31 usd: 163 [36531.065524] CPU 16: hi: 186, btch: 31 usd: 156 [36531.065527] CPU 17: hi: 186, btch: 31 usd: 183 [36531.065530] CPU 18: hi: 186, btch: 31 usd: 28 [36531.065533] CPU 19: hi: 186, btch: 31 usd: 160 [36531.065536] CPU 20: hi: 186, btch: 31 usd: 92 [36531.065539] CPU 21: hi: 186, btch: 31 usd: 160 [36531.065542] CPU 22: hi: 186, btch: 31 usd: 174 [36531.065545] CPU 23: hi: 186, btch: 31 usd: 152 [36531.065548] CPU 24: hi: 186, btch: 31 usd: 155 [36531.065550] CPU 25: hi: 186, btch: 31 usd: 164 [36531.065553] CPU 26: hi: 186, btch: 31 usd: 179 [36531.065556] CPU 27: hi: 186, btch: 31 usd: 174 [36531.065559] CPU 28: hi: 186, btch: 31 usd: 163 [36531.065562] CPU 29: hi: 186, btch: 31 usd: 92 [36531.065565] CPU 30: hi: 186, btch: 31 usd: 156 [36531.065568] CPU 31: hi: 186, btch: 31 usd: 164 [36531.065576] active_anon:164412 inactive_anon:26136 isolated_anon:0 [36531.065578] active_file:16696 inactive_file:15215036 isolated_file:32 [36531.065579] unevictable:2648 dirty:803 writeback:284 unstable:44711 [36531.065581] free:68369 slab_reclaimable:48276 slab_unreclaimable:483962 [36531.065582] mapped:12069 shmem:301 pagetables:2531 bounce:0 [36531.065586] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [36531.065602] lowmem_reserve[]: 0 2959 32249 32249 [36531.065608] Node 0 DMA32 free:123664kB min:4128kB low:5160kB high:6192kB active_anon:92kB inactive_anon:40964kB active_file:324kB inactive_file:1800952kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:332kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:39940kB slab_unreclaimable:454820kB kernel_stack:40kB pagetables:0kB unstable:24664kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no [36531.065625] lowmem_reserve[]: 0 0 29290 29290 [36531.065630] Node 0 Normal free:58788kB min:40864kB low:51080kB high:61296kB active_anon:461092kB inactive_anon:32660kB active_file:38404kB inactive_file:27226588kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:1368kB writeback:2264kB mapped:30476kB shmem:300kB slab_reclaimable:107864kB slab_unreclaimable:1067336kB kernel_stack:2304kB pagetables:4964kB unstable:112884kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [36531.065647] lowmem_reserve[]: 0 0 0 0 [36531.065652] Node 1 Normal free:75552kB min:45092kB low:56364kB high:67636kB active_anon:196464kB inactive_anon:30920kB active_file:28056kB inactive_file:31832220kB unevictable:2064kB isolated(anon):0kB isolated(file):128kB present:33095680kB mlocked:2064kB dirty:1512kB writeback:0kB mapped:17800kB shmem:904kB slab_reclaimable:45300kB slab_unreclaimable:413692kB kernel_stack:3016kB pagetables:5160kB unstable:41296kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [36531.065668] lowmem_reserve[]: 0 0 0 0 [36531.065672] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [36531.065685] Node 0 DMA32: 30650*4kB 108*8kB 10*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 123656kB [36531.065698] Node 0 Normal: 14388*4kB 311*8kB 11*16kB 7*32kB 11*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61144kB [36531.065714] Node 1 Normal: 14707*4kB 2162*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76172kB [36531.065727] 15232410 total pagecache pages [36531.065729] 0 pages in swap cache [36531.065732] Swap cache stats: add 0, delete 0, find 0/0 [36531.065735] Free swap = 8388604kB [36531.065737] Total swap = 8388604kB [36531.343534] 16777200 pages RAM [36531.343539] 393967 pages reserved [36531.343541] 1584160 pages shared [36531.343543] 14763235 pages non-shared [36531.343586] LNetError: 9246:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [36531.343594] LNetError: 9246:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315298434 total bytes allocated by lnet [36531.343604] LustreError: 9246:0:(niobuf.c:376:ptlrpc_register_bulk()) export-OST0007-osc-ffff88080dd47c00: LNetMDAttach failed x1456758786840756/0: rc = -12 [36531.343650] Lustre: 9246:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880cb2c83c00 x1456758786840756/t0(0) o4->export-OST0007-osc-ffff88080dd47c00@172.18.4.8@o2ib:6/4 lens 488/448 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [36531.343671] Lustre: export-OST0007-osc-ffff88080dd47c00: Connection to export-OST0007 (at 172.18.4.8@o2ib) was lost; in progress operations using this service will wait for recovery to complete [36531.370934] LustreError: 11-0: export-OST0007-osc-ffff88080dd47c00: Communicating with 172.18.4.8@o2ib, operation ost_connect failed with -16. [36556.329796] Lustre: export-OST0007-osc-ffff88080dd47c00: Connection restored to export-OST0007 (at 172.18.4.8@o2ib) [36985.763319] The following is only an harmless informational message. [36985.763326] Unless you get a _continuous_flood_ of these messages it means [36985.763329] everything is working fine. Allocations from irqs cannot be [36985.763332] perfectly reliable and the kernel is designed to handle that. [36985.763337] ptlrpcd_20: page allocation failure: order:1, mode:0x40 [36985.763343] Pid: 9235, comm: ptlrpcd_20 Tainted: P NX 3.0.51-0.7.9-default #1 [36985.763347] Call Trace: [36985.763376] [] dump_trace+0x75/0x300 [36985.763389] [] dump_stack+0x69/0x6f [36985.763400] [] warn_alloc_failed+0x102/0x1a0 [36985.763409] [] __alloc_pages_slowpath+0x541/0x7f0 [36985.763417] [] __alloc_pages_nodemask+0x1e9/0x200 [36985.763427] [] kmem_getpages+0x56/0x170 [36985.763435] [] fallback_alloc+0x19b/0x270 [36985.763442] [] __kmalloc+0x285/0x330 [36985.763521] [] ptlrpc_new_bulk+0x45/0x2a0 [ptlrpc] [36985.763642] [] ptlrpc_prep_bulk_imp+0x62/0x1b0 [ptlrpc] [36985.763692] [] osc_brw_prep_request+0x1bf/0xf20 [osc] [36985.763718] [] osc_build_rpc+0x7ba/0x1380 [osc] [36985.763747] [] osc_send_write_rpc+0x1e5/0x470 [osc] [36985.763789] [] osc_check_rpcs+0x5b4/0x710 [osc] [36985.763830] [] osc_io_unplug0+0x35b/0x4c0 [osc] [36985.763867] [] brw_queue_work+0x30/0xd0 [osc] [36985.763918] [] ptlrpc_check_set+0x522/0x1ae0 [ptlrpc] [36985.764003] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [36985.764105] [] ptlrpcd+0x32b/0x410 [ptlrpc] [36985.764166] [] kthread+0x96/0xa0 [36985.764176] [] kernel_thread_helper+0x4/0x10 [36985.764183] Mem-Info: [36985.764186] Node 0 DMA per-cpu: [36985.764190] CPU 0: hi: 0, btch: 1 usd: 0 [36985.764193] CPU 1: hi: 0, btch: 1 usd: 0 [36985.764197] CPU 2: hi: 0, btch: 1 usd: 0 [36985.764199] CPU 3: hi: 0, btch: 1 usd: 0 [36985.764202] CPU 4: hi: 0, btch: 1 usd: 0 [36985.764205] CPU 5: hi: 0, btch: 1 usd: 0 [36985.764208] CPU 6: hi: 0, btch: 1 usd: 0 [36985.764211] CPU 7: hi: 0, btch: 1 usd: 0 [36985.764214] CPU 8: hi: 0, btch: 1 usd: 0 [36985.764217] CPU 9: hi: 0, btch: 1 usd: 0 [36985.764220] CPU 10: hi: 0, btch: 1 usd: 0 [36985.764223] CPU 11: hi: 0, btch: 1 usd: 0 [36985.764226] CPU 12: hi: 0, btch: 1 usd: 0 [36985.764229] CPU 13: hi: 0, btch: 1 usd: 0 [36985.764231] CPU 14: hi: 0, btch: 1 usd: 0 [36985.764234] CPU 15: hi: 0, btch: 1 usd: 0 [36985.764237] CPU 16: hi: 0, btch: 1 usd: 0 [36985.764240] CPU 17: hi: 0, btch: 1 usd: 0 [36985.764243] CPU 18: hi: 0, btch: 1 usd: 0 [36985.764246] CPU 19: hi: 0, btch: 1 usd: 0 [36985.764249] CPU 20: hi: 0, btch: 1 usd: 0 [36985.764252] CPU 21: hi: 0, btch: 1 usd: 0 [36985.764255] CPU 22: hi: 0, btch: 1 usd: 0 [36985.764258] CPU 23: hi: 0, btch: 1 usd: 0 [36985.764261] CPU 24: hi: 0, btch: 1 usd: 0 [36985.764264] CPU 25: hi: 0, btch: 1 usd: 0 [36985.764266] CPU 26: hi: 0, btch: 1 usd: 0 [36985.764269] CPU 27: hi: 0, btch: 1 usd: 0 [36985.764272] CPU 28: hi: 0, btch: 1 usd: 0 [36985.764275] CPU 29: hi: 0, btch: 1 usd: 0 [36985.764278] CPU 30: hi: 0, btch: 1 usd: 0 [36985.764281] CPU 31: hi: 0, btch: 1 usd: 0 [36985.764283] Node 0 DMA32 per-cpu: [36985.764287] CPU 0: hi: 186, btch: 31 usd: 180 [36985.764291] CPU 1: hi: 186, btch: 31 usd: 180 [36985.764294] CPU 2: hi: 186, btch: 31 usd: 180 [36985.764297] CPU 3: hi: 186, btch: 31 usd: 31 [36985.764300] CPU 4: hi: 186, btch: 31 usd: 172 [36985.764303] CPU 5: hi: 186, btch: 31 usd: 183 [36985.764306] CPU 6: hi: 186, btch: 31 usd: 20 [36985.764309] CPU 7: hi: 186, btch: 31 usd: 33 [36985.764312] CPU 8: hi: 186, btch: 31 usd: 162 [36985.764315] CPU 9: hi: 186, btch: 31 usd: 0 [36985.764318] CPU 10: hi: 186, btch: 31 usd: 110 [36985.764321] CPU 11: hi: 186, btch: 31 usd: 145 [36985.764324] CPU 12: hi: 186, btch: 31 usd: 160 [36985.764327] CPU 13: hi: 186, btch: 31 usd: 180 [36985.764330] CPU 14: hi: 186, btch: 31 usd: 183 [36985.764333] CPU 15: hi: 186, btch: 31 usd: 165 [36985.764336] CPU 16: hi: 186, btch: 31 usd: 183 [36985.764339] CPU 17: hi: 186, btch: 31 usd: 181 [36985.764342] CPU 18: hi: 186, btch: 31 usd: 158 [36985.764345] CPU 19: hi: 186, btch: 31 usd: 170 [36985.764348] CPU 20: hi: 186, btch: 31 usd: 167 [36985.764351] CPU 21: hi: 186, btch: 31 usd: 156 [36985.764354] CPU 22: hi: 186, btch: 31 usd: 159 [36985.764357] CPU 23: hi: 186, btch: 31 usd: 171 [36985.764360] CPU 24: hi: 186, btch: 31 usd: 0 [36985.764363] CPU 25: hi: 186, btch: 31 usd: 158 [36985.764366] CPU 26: hi: 186, btch: 31 usd: 88 [36985.764369] CPU 27: hi: 186, btch: 31 usd: 0 [36985.764372] CPU 28: hi: 186, btch: 31 usd: 128 [36985.764375] CPU 29: hi: 186, btch: 31 usd: 162 [36985.764377] CPU 30: hi: 186, btch: 31 usd: 168 [36985.764380] CPU 31: hi: 186, btch: 31 usd: 35 [36985.764383] Node 0 Normal per-cpu: [36985.764386] CPU 0: hi: 186, btch: 31 usd: 185 [36985.764389] CPU 1: hi: 186, btch: 31 usd: 175 [36985.764392] CPU 2: hi: 186, btch: 31 usd: 172 [36985.764395] CPU 3: hi: 186, btch: 31 usd: 29 [36985.764398] CPU 4: hi: 186, btch: 31 usd: 162 [36985.764401] CPU 5: hi: 186, btch: 31 usd: 165 [36985.764404] CPU 6: hi: 186, btch: 31 usd: 75 [36985.764406] CPU 7: hi: 186, btch: 31 usd: 51 [36985.764409] CPU 8: hi: 186, btch: 31 usd: 179 [36985.764412] CPU 9: hi: 186, btch: 31 usd: 0 [36985.764415] CPU 10: hi: 186, btch: 31 usd: 174 [36985.764418] CPU 11: hi: 186, btch: 31 usd: 155 [36985.764421] CPU 12: hi: 186, btch: 31 usd: 167 [36985.764424] CPU 13: hi: 186, btch: 31 usd: 164 [36985.764426] CPU 14: hi: 186, btch: 31 usd: 157 [36985.764429] CPU 15: hi: 186, btch: 31 usd: 166 [36985.764432] CPU 16: hi: 186, btch: 31 usd: 163 [36985.764435] CPU 17: hi: 186, btch: 31 usd: 180 [36985.764438] CPU 18: hi: 186, btch: 31 usd: 156 [36985.764441] CPU 19: hi: 186, btch: 31 usd: 175 [36985.764444] CPU 20: hi: 186, btch: 31 usd: 161 [36985.764447] CPU 21: hi: 186, btch: 31 usd: 168 [36985.764449] CPU 22: hi: 186, btch: 31 usd: 174 [36985.764452] CPU 23: hi: 186, btch: 31 usd: 185 [36985.764455] CPU 24: hi: 186, btch: 31 usd: 0 [36985.764458] CPU 25: hi: 186, btch: 31 usd: 165 [36985.764461] CPU 26: hi: 186, btch: 31 usd: 0 [36985.764464] CPU 27: hi: 186, btch: 31 usd: 0 [36985.764466] CPU 28: hi: 186, btch: 31 usd: 19 [36985.764469] CPU 29: hi: 186, btch: 31 usd: 166 [36985.764472] CPU 30: hi: 186, btch: 31 usd: 180 [36985.764475] CPU 31: hi: 186, btch: 31 usd: 184 [36985.764478] Node 1 Normal per-cpu: [36985.764481] CPU 0: hi: 186, btch: 31 usd: 169 [36985.764484] CPU 1: hi: 186, btch: 31 usd: 170 [36985.764487] CPU 2: hi: 186, btch: 31 usd: 178 [36985.764490] CPU 3: hi: 186, btch: 31 usd: 33 [36985.764493] CPU 4: hi: 186, btch: 31 usd: 39 [36985.764496] CPU 5: hi: 186, btch: 31 usd: 176 [36985.764499] CPU 6: hi: 186, btch: 31 usd: 16 [36985.764502] CPU 7: hi: 186, btch: 31 usd: 30 [36985.764505] CPU 8: hi: 186, btch: 31 usd: 163 [36985.764508] CPU 9: hi: 186, btch: 31 usd: 178 [36985.764511] CPU 10: hi: 186, btch: 31 usd: 163 [36985.764514] CPU 11: hi: 186, btch: 31 usd: 168 [36985.764517] CPU 12: hi: 186, btch: 31 usd: 162 [36985.764520] CPU 13: hi: 186, btch: 31 usd: 163 [36985.764523] CPU 14: hi: 186, btch: 31 usd: 164 [36985.764526] CPU 15: hi: 186, btch: 31 usd: 157 [36985.764529] CPU 16: hi: 186, btch: 31 usd: 169 [36985.764532] CPU 17: hi: 186, btch: 31 usd: 179 [36985.764535] CPU 18: hi: 186, btch: 31 usd: 52 [36985.764538] CPU 19: hi: 186, btch: 31 usd: 26 [36985.764541] CPU 20: hi: 186, btch: 31 usd: 116 [36985.764544] CPU 21: hi: 186, btch: 31 usd: 143 [36985.764547] CPU 22: hi: 186, btch: 31 usd: 184 [36985.764550] CPU 23: hi: 186, btch: 31 usd: 167 [36985.764553] CPU 24: hi: 186, btch: 31 usd: 156 [36985.764556] CPU 25: hi: 186, btch: 31 usd: 174 [36985.764559] CPU 26: hi: 186, btch: 31 usd: 154 [36985.764562] CPU 27: hi: 186, btch: 31 usd: 154 [36985.764565] CPU 28: hi: 186, btch: 31 usd: 171 [36985.764568] CPU 29: hi: 186, btch: 31 usd: 165 [36985.764570] CPU 30: hi: 186, btch: 31 usd: 176 [36985.764573] CPU 31: hi: 186, btch: 31 usd: 158 [36985.764581] active_anon:175562 inactive_anon:26136 isolated_anon:0 [36985.764583] active_file:17291 inactive_file:15242308 isolated_file:64 [36985.764584] unevictable:2648 dirty:651 writeback:769 unstable:66624 [36985.764586] free:62950 slab_reclaimable:48437 slab_unreclaimable:450619 [36985.764588] mapped:12199 shmem:301 pagetables:2785 bounce:0 [36985.764591] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [36985.764607] lowmem_reserve[]: 0 2959 32249 32249 [36985.764613] Node 0 DMA32 free:124244kB min:4128kB low:5160kB high:6192kB active_anon:36kB inactive_anon:40964kB active_file:140kB inactive_file:1736520kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:0kB writeback:1744kB mapped:0kB shmem:0kB slab_reclaimable:39244kB slab_unreclaimable:519960kB kernel_stack:40kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no [36985.764630] lowmem_reserve[]: 0 0 29290 29290 [36985.764634] Node 0 Normal free:52048kB min:40864kB low:51080kB high:61296kB active_anon:502884kB inactive_anon:32660kB active_file:37896kB inactive_file:27123404kB unevictable:8528kB isolated(anon):0kB isolated(file):128kB present:29992960kB mlocked:2396kB dirty:1056kB writeback:188kB mapped:30820kB shmem:300kB slab_reclaimable:106644kB slab_unreclaimable:1139036kB kernel_stack:2328kB pagetables:5624kB unstable:218412kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no [36985.764652] lowmem_reserve[]: 0 0 0 0 [36985.764657] Node 1 Normal free:60036kB min:45092kB low:56364kB high:67636kB active_anon:199328kB inactive_anon:30920kB active_file:31128kB inactive_file:32108924kB unevictable:2064kB isolated(anon):0kB isolated(file):128kB present:33095680kB mlocked:2064kB dirty:1548kB writeback:1144kB mapped:17976kB shmem:904kB slab_reclaimable:47860kB slab_unreclaimable:143480kB kernel_stack:3056kB pagetables:5516kB unstable:48084kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [36985.764674] lowmem_reserve[]: 0 0 0 0 [36985.764678] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [36985.764691] Node 0 DMA32: 30988*4kB 47*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 124360kB [36985.764704] Node 0 Normal: 12896*4kB 188*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 53104kB [36985.764716] Node 1 Normal: 14167*4kB 499*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60740kB [36985.764729] 15260146 total pagecache pages [36985.764731] 0 pages in swap cache [36985.764734] Swap cache stats: add 0, delete 0, find 0/0 [36985.764737] Free swap = 8388604kB [36985.764739] Total swap = 8388604kB [36986.047322] 16777200 pages RAM [36986.047326] 393967 pages reserved [36986.047329] 1607644 pages shared [36986.047331] 14763263 pages non-shared [36986.047400] LustreError: 9235:0:(osc_request.c:2198:osc_build_rpc()) prep_req failed: -12 [36986.047912] LustreError: 9235:0:(osc_cache.c:2211:osc_check_rpcs()) Write request failed with -12 [37160.676118] The following is only an harmless informational message. [37160.676125] Unless you get a _continuous_flood_ of these messages it means [37160.676127] everything is working fine. Allocations from irqs cannot be [37160.676130] perfectly reliable and the kernel is designed to handle that. [37160.676135] ptlrpcd_24: page allocation failure: order:1, mode:0x40 [37160.676141] Pid: 9239, comm: ptlrpcd_24 Tainted: P NX 3.0.51-0.7.9-default #1 [37160.676146] Call Trace: [37160.676172] [] dump_trace+0x75/0x300 [37160.676185] [] dump_stack+0x69/0x6f [37160.676196] [] warn_alloc_failed+0x102/0x1a0 [37160.676205] [] __alloc_pages_slowpath+0x541/0x7f0 [37160.676213] [] __alloc_pages_nodemask+0x1e9/0x200 [37160.676223] [] kmem_getpages+0x56/0x170 [37160.676231] [] fallback_alloc+0x19b/0x270 [37160.676238] [] __kmalloc+0x285/0x330 [37160.676316] [] ptlrpc_new_bulk+0x45/0x2a0 [ptlrpc] [37160.676417] [] ptlrpc_prep_bulk_imp+0x62/0x1b0 [ptlrpc] [37160.676465] [] osc_brw_prep_request+0x1bf/0xf20 [osc] [37160.676490] [] osc_build_rpc+0x7ba/0x1380 [osc] [37160.676519] [] osc_send_write_rpc+0x1e5/0x470 [osc] [37160.676561] [] osc_check_rpcs+0x5b4/0x710 [osc] [37160.676602] [] osc_io_unplug0+0x35b/0x4c0 [osc] [37160.676638] [] brw_queue_work+0x30/0xd0 [osc] [37160.676689] [] ptlrpc_check_set+0x522/0x1ae0 [ptlrpc] [37160.676793] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [37160.676905] [] ptlrpcd+0x32b/0x410 [ptlrpc] [37160.676964] [] kthread+0x96/0xa0 [37160.676974] [] kernel_thread_helper+0x4/0x10 [37160.676981] Mem-Info: [37160.676984] Node 0 DMA per-cpu: [37160.676988] CPU 0: hi: 0, btch: 1 usd: 0 [37160.676991] CPU 1: hi: 0, btch: 1 usd: 0 [37160.676994] CPU 2: hi: 0, btch: 1 usd: 0 [37160.676997] CPU 3: hi: 0, btch: 1 usd: 0 [37160.677000] CPU 4: hi: 0, btch: 1 usd: 0 [37160.677003] CPU 5: hi: 0, btch: 1 usd: 0 [37160.677006] CPU 6: hi: 0, btch: 1 usd: 0 [37160.677009] CPU 7: hi: 0, btch: 1 usd: 0 [37160.677012] CPU 8: hi: 0, btch: 1 usd: 0 [37160.677014] CPU 9: hi: 0, btch: 1 usd: 0 [37160.677018] CPU 10: hi: 0, btch: 1 usd: 0 [37160.677020] CPU 11: hi: 0, btch: 1 usd: 0 [37160.677023] CPU 12: hi: 0, btch: 1 usd: 0 [37160.677026] CPU 13: hi: 0, btch: 1 usd: 0 [37160.677029] CPU 14: hi: 0, btch: 1 usd: 0 [37160.677032] CPU 15: hi: 0, btch: 1 usd: 0 [37160.677035] CPU 16: hi: 0, btch: 1 usd: 0 [37160.677038] CPU 17: hi: 0, btch: 1 usd: 0 [37160.677041] CPU 18: hi: 0, btch: 1 usd: 0 [37160.677044] CPU 19: hi: 0, btch: 1 usd: 0 [37160.677047] CPU 20: hi: 0, btch: 1 usd: 0 [37160.677050] CPU 21: hi: 0, btch: 1 usd: 0 [37160.677053] CPU 22: hi: 0, btch: 1 usd: 0 [37160.677056] CPU 23: hi: 0, btch: 1 usd: 0 [37160.677058] CPU 24: hi: 0, btch: 1 usd: 0 [37160.677061] CPU 25: hi: 0, btch: 1 usd: 0 [37160.677064] CPU 26: hi: 0, btch: 1 usd: 0 [37160.677067] CPU 27: hi: 0, btch: 1 usd: 0 [37160.677070] CPU 28: hi: 0, btch: 1 usd: 0 [37160.677073] CPU 29: hi: 0, btch: 1 usd: 0 [37160.677076] CPU 30: hi: 0, btch: 1 usd: 0 [37160.677078] CPU 31: hi: 0, btch: 1 usd: 0 [37160.677081] Node 0 DMA32 per-cpu: [37160.677085] CPU 0: hi: 186, btch: 31 usd: 34 [37160.677088] CPU 1: hi: 186, btch: 31 usd: 59 [37160.677091] CPU 2: hi: 186, btch: 31 usd: 173 [37160.677094] CPU 3: hi: 186, btch: 31 usd: 167 [37160.677097] CPU 4: hi: 186, btch: 31 usd: 23 [37160.677100] CPU 5: hi: 186, btch: 31 usd: 0 [37160.677103] CPU 6: hi: 186, btch: 31 usd: 159 [37160.677106] CPU 7: hi: 186, btch: 31 usd: 174 [37160.677109] CPU 8: hi: 186, btch: 31 usd: 172 [37160.677112] CPU 9: hi: 186, btch: 31 usd: 14 [37160.677115] CPU 10: hi: 186, btch: 31 usd: 97 [37160.677118] CPU 11: hi: 186, btch: 31 usd: 135 [37160.677121] CPU 12: hi: 186, btch: 31 usd: 135 [37160.677124] CPU 13: hi: 186, btch: 31 usd: 137 [37160.677127] CPU 14: hi: 186, btch: 31 usd: 138 [37160.677129] CPU 15: hi: 186, btch: 31 usd: 162 [37160.677132] CPU 16: hi: 186, btch: 31 usd: 161 [37160.677135] CPU 17: hi: 186, btch: 31 usd: 177 [37160.677138] CPU 18: hi: 186, btch: 31 usd: 170 [37160.677141] CPU 19: hi: 186, btch: 31 usd: 168 [37160.677145] CPU 20: hi: 186, btch: 31 usd: 160 [37160.677148] CPU 21: hi: 186, btch: 31 usd: 163 [37160.677151] CPU 22: hi: 186, btch: 31 usd: 132 [37160.677154] CPU 23: hi: 186, btch: 31 usd: 163 [37160.677157] CPU 24: hi: 186, btch: 31 usd: 166 [37160.677159] CPU 25: hi: 186, btch: 31 usd: 174 [37160.677162] CPU 26: hi: 186, btch: 31 usd: 88 [37160.677165] CPU 27: hi: 186, btch: 31 usd: 0 [37160.677168] CPU 28: hi: 186, btch: 31 usd: 135 [37160.677171] CPU 29: hi: 186, btch: 31 usd: 168 [37160.677174] CPU 30: hi: 186, btch: 31 usd: 115 [37160.677177] CPU 31: hi: 186, btch: 31 usd: 45 [37160.677179] Node 0 Normal per-cpu: [37160.677183] CPU 0: hi: 186, btch: 31 usd: 26 [37160.677186] CPU 1: hi: 186, btch: 31 usd: 19 [37160.677189] CPU 2: hi: 186, btch: 31 usd: 171 [37160.677192] CPU 3: hi: 186, btch: 31 usd: 171 [37160.677195] CPU 4: hi: 186, btch: 31 usd: 166 [37160.677198] CPU 5: hi: 186, btch: 31 usd: 16 [37160.677201] CPU 6: hi: 186, btch: 31 usd: 158 [37160.677203] CPU 7: hi: 186, btch: 31 usd: 173 [37160.677206] CPU 8: hi: 186, btch: 31 usd: 157 [37160.677209] CPU 9: hi: 186, btch: 31 usd: 0 [37160.677212] CPU 10: hi: 186, btch: 31 usd: 181 [37160.677215] CPU 11: hi: 186, btch: 31 usd: 180 [37160.677218] CPU 12: hi: 186, btch: 31 usd: 179 [37160.677220] CPU 13: hi: 186, btch: 31 usd: 175 [37160.677223] CPU 14: hi: 186, btch: 31 usd: 174 [37160.677226] CPU 15: hi: 186, btch: 31 usd: 161 [37160.677229] CPU 16: hi: 186, btch: 31 usd: 176 [37160.677232] CPU 17: hi: 186, btch: 31 usd: 166 [37160.677235] CPU 18: hi: 186, btch: 31 usd: 135 [37160.677238] CPU 19: hi: 186, btch: 31 usd: 163 [37160.677241] CPU 20: hi: 186, btch: 31 usd: 181 [37160.677244] CPU 21: hi: 186, btch: 31 usd: 11 [37160.677246] CPU 22: hi: 186, btch: 31 usd: 164 [37160.677249] CPU 23: hi: 186, btch: 31 usd: 173 [37160.677252] CPU 24: hi: 186, btch: 31 usd: 166 [37160.677255] CPU 25: hi: 186, btch: 31 usd: 172 [37160.677258] CPU 26: hi: 186, btch: 31 usd: 0 [37160.677261] CPU 27: hi: 186, btch: 31 usd: 0 [37160.677264] CPU 28: hi: 186, btch: 31 usd: 174 [37160.677267] CPU 29: hi: 186, btch: 31 usd: 183 [37160.677270] CPU 30: hi: 186, btch: 31 usd: 160 [37160.677273] CPU 31: hi: 186, btch: 31 usd: 180 [37160.677275] Node 1 Normal per-cpu: [37160.677279] CPU 0: hi: 186, btch: 31 usd: 18 [37160.677282] CPU 1: hi: 186, btch: 31 usd: 25 [37160.677285] CPU 2: hi: 186, btch: 31 usd: 170 [37160.677288] CPU 3: hi: 186, btch: 31 usd: 154 [37160.677291] CPU 4: hi: 186, btch: 31 usd: 0 [37160.677294] CPU 5: hi: 186, btch: 31 usd: 0 [37160.677297] CPU 6: hi: 186, btch: 31 usd: 163 [37160.677300] CPU 7: hi: 186, btch: 31 usd: 161 [37160.677303] CPU 8: hi: 186, btch: 31 usd: 168 [37160.677306] CPU 9: hi: 186, btch: 31 usd: 56 [37160.677309] CPU 10: hi: 186, btch: 31 usd: 197 [37160.677312] CPU 11: hi: 186, btch: 31 usd: 180 [37160.677314] CPU 12: hi: 186, btch: 31 usd: 166 [37160.677317] CPU 13: hi: 186, btch: 31 usd: 159 [37160.677320] CPU 14: hi: 186, btch: 31 usd: 178 [37160.677323] CPU 15: hi: 186, btch: 31 usd: 161 [37160.677328] CPU 16: hi: 186, btch: 31 usd: 156 [37160.677331] CPU 17: hi: 186, btch: 31 usd: 182 [37160.677334] CPU 18: hi: 186, btch: 31 usd: 158 [37160.677337] CPU 19: hi: 186, btch: 31 usd: 94 [37160.677340] CPU 20: hi: 186, btch: 31 usd: 156 [37160.677343] CPU 21: hi: 186, btch: 31 usd: 15 [37160.677346] CPU 22: hi: 186, btch: 31 usd: 0 [37160.677349] CPU 23: hi: 186, btch: 31 usd: 185 [37160.677352] CPU 24: hi: 186, btch: 31 usd: 167 [37160.677355] CPU 25: hi: 186, btch: 31 usd: 178 [37160.677358] CPU 26: hi: 186, btch: 31 usd: 161 [37160.677361] CPU 27: hi: 186, btch: 31 usd: 154 [37160.677364] CPU 28: hi: 186, btch: 31 usd: 160 [37160.677367] CPU 29: hi: 186, btch: 31 usd: 175 [37160.677370] CPU 30: hi: 186, btch: 31 usd: 182 [37160.677373] CPU 31: hi: 186, btch: 31 usd: 177 [37160.677381] active_anon:175644 inactive_anon:26136 isolated_anon:0 [37160.677383] active_file:16679 inactive_file:15234084 isolated_file:32 [37160.677384] unevictable:2648 dirty:412 writeback:636 unstable:65222 [37160.677386] free:69856 slab_reclaimable:48839 slab_unreclaimable:452150 [37160.677388] mapped:12209 shmem:301 pagetables:2792 bounce:0 [37160.677391] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [37160.677407] lowmem_reserve[]: 0 2959 32249 32249 [37160.677413] Node 0 DMA32 free:123440kB min:4128kB low:5160kB high:6192kB active_anon:88kB inactive_anon:40964kB active_file:24kB inactive_file:1827148kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:112kB writeback:644kB mapped:0kB shmem:0kB slab_reclaimable:38960kB slab_unreclaimable:430596kB kernel_stack:40kB pagetables:0kB unstable:1880kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37160.677430] lowmem_reserve[]: 0 0 29290 29290 [37160.677435] Node 0 Normal free:64348kB min:40864kB low:51080kB high:61296kB active_anon:501992kB inactive_anon:32660kB active_file:36716kB inactive_file:27157096kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:476kB writeback:1212kB mapped:30820kB shmem:300kB slab_reclaimable:106448kB slab_unreclaimable:1094648kB kernel_stack:2328kB pagetables:5624kB unstable:202824kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no [37160.677453] lowmem_reserve[]: 0 0 0 0 [37160.677457] Node 1 Normal free:75792kB min:45092kB low:56364kB high:67636kB active_anon:200496kB inactive_anon:30920kB active_file:29976kB inactive_file:31952092kB unevictable:2064kB isolated(anon):0kB isolated(file):128kB present:33095680kB mlocked:2064kB dirty:1060kB writeback:688kB mapped:18016kB shmem:904kB slab_reclaimable:49948kB slab_unreclaimable:283356kB kernel_stack:3064kB pagetables:5544kB unstable:56184kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37160.677474] lowmem_reserve[]: 0 0 0 0 [37160.677479] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [37160.677492] Node 0 DMA32: 30798*4kB 91*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 123920kB [37160.677505] Node 0 Normal: 12871*4kB 1668*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64844kB [37160.677517] Node 1 Normal: 14693*4kB 2121*8kB 41*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76396kB [37160.677530] 15251771 total pagecache pages [37160.677532] 0 pages in swap cache [37160.677535] Swap cache stats: add 0, delete 0, find 0/0 [37160.677538] Free swap = 8388604kB [37160.677540] Total swap = 8388604kB [37160.954901] 16777200 pages RAM [37160.954905] 393967 pages reserved [37160.954907] 1609038 pages shared [37160.954909] 14766716 pages non-shared [37160.954969] LustreError: 9239:0:(osc_request.c:2198:osc_build_rpc()) prep_req failed: -12 [37160.955330] LustreError: 9239:0:(osc_cache.c:2211:osc_check_rpcs()) Write request failed with -12 [37681.958983] The following is only an harmless informational message. [37681.958988] Unless you get a _continuous_flood_ of these messages it means [37681.958989] everything is working fine. Allocations from irqs cannot be [37681.958991] perfectly reliable and the kernel is designed to handle that. [37681.958994] ptlrpcd_6: page allocation failure: order:1, mode:0x40 [37681.958997] Pid: 9221, comm: ptlrpcd_6 Tainted: P NX 3.0.51-0.7.9-default #1 [37681.958999] Call Trace: [37681.959015] [] dump_trace+0x75/0x300 [37681.959022] [] dump_stack+0x69/0x6f [37681.959028] [] warn_alloc_failed+0x102/0x1a0 [37681.959032] [] __alloc_pages_slowpath+0x541/0x7f0 [37681.959037] [] __alloc_pages_nodemask+0x1e9/0x200 [37681.959042] [] kmem_getpages+0x56/0x170 [37681.959046] [] fallback_alloc+0x19b/0x270 [37681.959050] [] __kmalloc+0x285/0x330 [37681.959067] [] LNetMDAttach+0x163/0x5b0 [lnet] [37681.959131] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [37681.959174] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [37681.959216] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [37681.959256] [] ptlrpc_check_set+0x408/0x1ae0 [ptlrpc] [37681.959302] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [37681.959387] [] ptlrpcd+0x32b/0x410 [ptlrpc] [37681.959445] [] kthread+0x96/0xa0 [37681.959454] [] kernel_thread_helper+0x4/0x10 [37681.959460] Mem-Info: [37681.959462] Node 0 DMA per-cpu: [37681.959466] CPU 0: hi: 0, btch: 1 usd: 0 [37681.959469] CPU 1: hi: 0, btch: 1 usd: 0 [37681.959472] CPU 2: hi: 0, btch: 1 usd: 0 [37681.959475] CPU 3: hi: 0, btch: 1 usd: 0 [37681.959478] CPU 4: hi: 0, btch: 1 usd: 0 [37681.959481] CPU 5: hi: 0, btch: 1 usd: 0 [37681.959484] CPU 6: hi: 0, btch: 1 usd: 0 [37681.959487] CPU 7: hi: 0, btch: 1 usd: 0 [37681.959490] CPU 8: hi: 0, btch: 1 usd: 0 [37681.959493] CPU 9: hi: 0, btch: 1 usd: 0 [37681.959496] CPU 10: hi: 0, btch: 1 usd: 0 [37681.959499] CPU 11: hi: 0, btch: 1 usd: 0 [37681.959502] CPU 12: hi: 0, btch: 1 usd: 0 [37681.959505] CPU 13: hi: 0, btch: 1 usd: 0 [37681.959508] CPU 14: hi: 0, btch: 1 usd: 0 [37681.959511] CPU 15: hi: 0, btch: 1 usd: 0 [37681.959514] CPU 16: hi: 0, btch: 1 usd: 0 [37681.959516] CPU 17: hi: 0, btch: 1 usd: 0 [37681.959519] CPU 18: hi: 0, btch: 1 usd: 0 [37681.959522] CPU 19: hi: 0, btch: 1 usd: 0 [37681.959525] CPU 20: hi: 0, btch: 1 usd: 0 [37681.959528] CPU 21: hi: 0, btch: 1 usd: 0 [37681.959531] CPU 22: hi: 0, btch: 1 usd: 0 [37681.959534] CPU 23: hi: 0, btch: 1 usd: 0 [37681.959537] CPU 24: hi: 0, btch: 1 usd: 0 [37681.959540] CPU 25: hi: 0, btch: 1 usd: 0 [37681.959543] CPU 26: hi: 0, btch: 1 usd: 0 [37681.959546] CPU 27: hi: 0, btch: 1 usd: 0 [37681.959548] CPU 28: hi: 0, btch: 1 usd: 0 [37681.959551] CPU 29: hi: 0, btch: 1 usd: 0 [37681.959554] CPU 30: hi: 0, btch: 1 usd: 0 [37681.959557] CPU 31: hi: 0, btch: 1 usd: 0 [37681.959560] Node 0 DMA32 per-cpu: [37681.959563] CPU 0: hi: 186, btch: 31 usd: 31 [37681.959566] CPU 1: hi: 186, btch: 31 usd: 32 [37681.959569] CPU 2: hi: 186, btch: 31 usd: 157 [37681.959572] CPU 3: hi: 186, btch: 31 usd: 12 [37681.959575] CPU 4: hi: 186, btch: 31 usd: 13 [37681.959578] CPU 5: hi: 186, btch: 31 usd: 170 [37681.959581] CPU 6: hi: 186, btch: 31 usd: 182 [37681.959584] CPU 7: hi: 186, btch: 31 usd: 170 [37681.959587] CPU 8: hi: 186, btch: 31 usd: 78 [37681.959590] CPU 9: hi: 186, btch: 31 usd: 159 [37681.959593] CPU 10: hi: 186, btch: 31 usd: 165 [37681.959596] CPU 11: hi: 186, btch: 31 usd: 67 [37681.959599] CPU 12: hi: 186, btch: 31 usd: 182 [37681.959603] CPU 13: hi: 186, btch: 31 usd: 0 [37681.959606] CPU 14: hi: 186, btch: 31 usd: 179 [37681.959609] CPU 15: hi: 186, btch: 31 usd: 146 [37681.959612] CPU 16: hi: 186, btch: 31 usd: 158 [37681.959615] CPU 17: hi: 186, btch: 31 usd: 161 [37681.959617] CPU 18: hi: 186, btch: 31 usd: 157 [37681.959620] CPU 19: hi: 186, btch: 31 usd: 167 [37681.959623] CPU 20: hi: 186, btch: 31 usd: 178 [37681.959626] CPU 21: hi: 186, btch: 31 usd: 177 [37681.959629] CPU 22: hi: 186, btch: 31 usd: 172 [37681.959632] CPU 23: hi: 186, btch: 31 usd: 170 [37681.959635] CPU 24: hi: 186, btch: 31 usd: 57 [37681.959638] CPU 25: hi: 186, btch: 31 usd: 128 [37681.959641] CPU 26: hi: 186, btch: 31 usd: 115 [37681.959644] CPU 27: hi: 186, btch: 31 usd: 173 [37681.959647] CPU 28: hi: 186, btch: 31 usd: 127 [37681.959650] CPU 29: hi: 186, btch: 31 usd: 161 [37681.959653] CPU 30: hi: 186, btch: 31 usd: 148 [37681.959656] CPU 31: hi: 186, btch: 31 usd: 181 [37681.959659] Node 0 Normal per-cpu: [37681.959662] CPU 0: hi: 186, btch: 31 usd: 200 [37681.959665] CPU 1: hi: 186, btch: 31 usd: 38 [37681.959668] CPU 2: hi: 186, btch: 31 usd: 175 [37681.959670] CPU 3: hi: 186, btch: 31 usd: 156 [37681.959673] CPU 4: hi: 186, btch: 31 usd: 25 [37681.959676] CPU 5: hi: 186, btch: 31 usd: 165 [37681.959679] CPU 6: hi: 186, btch: 31 usd: 170 [37681.959682] CPU 7: hi: 186, btch: 31 usd: 214 [37681.959685] CPU 8: hi: 186, btch: 31 usd: 0 [37681.959688] CPU 9: hi: 186, btch: 31 usd: 159 [37681.959690] CPU 10: hi: 186, btch: 31 usd: 185 [37681.959693] CPU 11: hi: 186, btch: 31 usd: 140 [37681.959708] CPU 12: hi: 186, btch: 31 usd: 179 [37681.959709] CPU 13: hi: 186, btch: 31 usd: 1 [37681.959711] CPU 14: hi: 186, btch: 31 usd: 159 [37681.959712] CPU 15: hi: 186, btch: 31 usd: 162 [37681.959714] CPU 16: hi: 186, btch: 31 usd: 173 [37681.959715] CPU 17: hi: 186, btch: 31 usd: 48 [37681.959717] CPU 18: hi: 186, btch: 31 usd: 160 [37681.959719] CPU 19: hi: 186, btch: 31 usd: 180 [37681.959720] CPU 20: hi: 186, btch: 31 usd: 168 [37681.959722] CPU 21: hi: 186, btch: 31 usd: 173 [37681.959723] CPU 22: hi: 186, btch: 31 usd: 162 [37681.959725] CPU 23: hi: 186, btch: 31 usd: 178 [37681.959726] CPU 24: hi: 186, btch: 31 usd: 1 [37681.959728] CPU 25: hi: 186, btch: 31 usd: 142 [37681.959730] CPU 26: hi: 186, btch: 31 usd: 98 [37681.959731] CPU 27: hi: 186, btch: 31 usd: 165 [37681.959733] CPU 28: hi: 186, btch: 31 usd: 13 [37681.959734] CPU 29: hi: 186, btch: 31 usd: 178 [37681.959736] CPU 30: hi: 186, btch: 31 usd: 166 [37681.959738] CPU 31: hi: 186, btch: 31 usd: 162 [37681.959739] Node 1 Normal per-cpu: [37681.959741] CPU 0: hi: 186, btch: 31 usd: 27 [37681.959742] CPU 1: hi: 186, btch: 31 usd: 10 [37681.959744] CPU 2: hi: 186, btch: 31 usd: 184 [37681.959746] CPU 3: hi: 186, btch: 31 usd: 34 [37681.959747] CPU 4: hi: 186, btch: 31 usd: 0 [37681.959749] CPU 5: hi: 186, btch: 31 usd: 164 [37681.959750] CPU 6: hi: 186, btch: 31 usd: 178 [37681.959752] CPU 7: hi: 186, btch: 31 usd: 183 [37681.959754] CPU 8: hi: 186, btch: 31 usd: 180 [37681.959755] CPU 9: hi: 186, btch: 31 usd: 176 [37681.959757] CPU 10: hi: 186, btch: 31 usd: 171 [37681.959759] CPU 11: hi: 186, btch: 31 usd: 171 [37681.959761] CPU 12: hi: 186, btch: 31 usd: 154 [37681.959762] CPU 13: hi: 186, btch: 31 usd: 50 [37681.959764] CPU 14: hi: 186, btch: 31 usd: 163 [37681.959766] CPU 15: hi: 186, btch: 31 usd: 165 [37681.959767] CPU 16: hi: 186, btch: 31 usd: 169 [37681.959769] CPU 17: hi: 186, btch: 31 usd: 34 [37681.959770] CPU 18: hi: 186, btch: 31 usd: 175 [37681.959772] CPU 19: hi: 186, btch: 31 usd: 183 [37681.959774] CPU 20: hi: 186, btch: 31 usd: 155 [37681.959775] CPU 21: hi: 186, btch: 31 usd: 176 [37681.959777] CPU 22: hi: 186, btch: 31 usd: 155 [37681.959779] CPU 23: hi: 186, btch: 31 usd: 137 [37681.959780] CPU 24: hi: 186, btch: 31 usd: 182 [37681.959782] CPU 25: hi: 186, btch: 31 usd: 176 [37681.959784] CPU 26: hi: 186, btch: 31 usd: 161 [37681.959786] CPU 27: hi: 186, btch: 31 usd: 176 [37681.959787] CPU 28: hi: 186, btch: 31 usd: 171 [37681.959789] CPU 29: hi: 186, btch: 31 usd: 163 [37681.959791] CPU 30: hi: 186, btch: 31 usd: 177 [37681.959792] CPU 31: hi: 186, btch: 31 usd: 162 [37681.959797] active_anon:175647 inactive_anon:26136 isolated_anon:0 [37681.959798] active_file:19192 inactive_file:14808831 isolated_file:32 [37681.959799] unevictable:2648 dirty:1224 writeback:0 unstable:122321 [37681.959799] free:63129 slab_reclaimable:51948 slab_unreclaimable:878494 [37681.959800] mapped:12206 shmem:301 pagetables:2794 bounce:0 [37681.959802] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [37681.959811] lowmem_reserve[]: 0 2959 32249 32249 [37681.959814] Node 0 DMA32 free:122392kB min:4128kB low:5160kB high:6192kB active_anon:36kB inactive_anon:40964kB active_file:136kB inactive_file:1413400kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:284kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:39420kB slab_unreclaimable:844368kB kernel_stack:40kB pagetables:0kB unstable:33380kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37681.959823] lowmem_reserve[]: 0 0 29290 29290 [37681.959826] Node 0 Normal free:52972kB min:40864kB low:51080kB high:61296kB active_anon:501936kB inactive_anon:32660kB active_file:45036kB inactive_file:26705032kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:3096kB writeback:188kB mapped:30808kB shmem:300kB slab_reclaimable:108612kB slab_unreclaimable:1549232kB kernel_stack:2328kB pagetables:5624kB unstable:333852kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37681.959835] lowmem_reserve[]: 0 0 0 0 [37681.959838] Node 1 Normal free:61308kB min:45092kB low:56364kB high:67636kB active_anon:200616kB inactive_anon:30920kB active_file:31596kB inactive_file:31116508kB unevictable:2064kB isolated(anon):0kB isolated(file):128kB present:33095680kB mlocked:2064kB dirty:1516kB writeback:0kB mapped:18016kB shmem:904kB slab_reclaimable:59760kB slab_unreclaimable:1120376kB kernel_stack:3072kB pagetables:5552kB unstable:122052kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37681.959847] lowmem_reserve[]: 0 0 0 0 [37681.959849] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [37681.959856] Node 0 DMA32: 30671*4kB 31*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 122932kB [37681.959863] Node 0 Normal: 13161*4kB 103*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 53468kB [37681.959870] Node 1 Normal: 13996*4kB 466*8kB 113*16kB 5*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61680kB [37681.959880] 14828900 total pagecache pages [37681.959881] 0 pages in swap cache [37681.959883] Swap cache stats: add 0, delete 0, find 0/0 [37681.959884] Free swap = 8388604kB [37681.959885] Total swap = 8388604kB [37682.199154] 16777200 pages RAM [37682.199156] 393967 pages reserved [37682.199157] 3310806 pages shared [37682.199158] 13088033 pages non-shared [37682.199181] LNetError: 9221:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [37682.199187] LNetError: 9221:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315259714 total bytes allocated by lnet [37682.199192] LustreError: 9221:0:(niobuf.c:376:ptlrpc_register_bulk()) export-OST0009-osc-ffff88080dd47c00: LNetMDAttach failed x1456758787630940/0: rc = -12 [37682.199214] Lustre: 9221:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880349b69800 x1456758787630940/t0(0) o4->export-OST0009-osc-ffff88080dd47c00@172.18.4.8@o2ib:6/4 lens 488/448 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [37682.199229] Lustre: export-OST0009-osc-ffff88080dd47c00: Connection to export-OST0009 (at 172.18.4.8@o2ib) was lost; in progress operations using this service will wait for recovery to complete [37682.210314] Lustre: export-OST0009-osc-ffff88080dd47c00: Connection restored to export-OST0009 (at 172.18.4.8@o2ib) [37689.551784] The following is only an harmless informational message. [37689.551789] Unless you get a _continuous_flood_ of these messages it means [37689.551792] everything is working fine. Allocations from irqs cannot be [37689.551794] perfectly reliable and the kernel is designed to handle that. [37689.551797] ptlrpcd_19: page allocation failure: order:1, mode:0x40 [37689.551802] Pid: 9234, comm: ptlrpcd_19 Tainted: P NX 3.0.51-0.7.9-default #1 [37689.551805] Call Trace: [37689.551824] [] dump_trace+0x75/0x300 [37689.551832] [] dump_stack+0x69/0x6f [37689.551838] [] warn_alloc_failed+0x102/0x1a0 [37689.551843] [] __alloc_pages_slowpath+0x541/0x7f0 [37689.551849] [] __alloc_pages_nodemask+0x1e9/0x200 [37689.551857] [] kmem_getpages+0x56/0x170 [37689.551864] [] fallback_alloc+0x19b/0x270 [37689.551870] [] __kmalloc+0x285/0x330 [37689.551890] [] LNetMDAttach+0x163/0x5b0 [lnet] [37689.552025] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [37689.552157] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [37689.552284] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [37689.552405] [] ptlrpc_check_set+0x408/0x1ae0 [ptlrpc] [37689.552532] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [37689.552699] [] ptlrpcd+0x32b/0x410 [ptlrpc] [37689.552807] [] kthread+0x96/0xa0 [37689.552821] [] kernel_thread_helper+0x4/0x10 [37689.552831] Mem-Info: [37689.552835] Node 0 DMA per-cpu: [37689.552841] CPU 0: hi: 0, btch: 1 usd: 0 [37689.552847] CPU 1: hi: 0, btch: 1 usd: 0 [37689.552852] CPU 2: hi: 0, btch: 1 usd: 0 [37689.552857] CPU 3: hi: 0, btch: 1 usd: 0 [37689.552862] CPU 4: hi: 0, btch: 1 usd: 0 [37689.552867] CPU 5: hi: 0, btch: 1 usd: 0 [37689.552872] CPU 6: hi: 0, btch: 1 usd: 0 [37689.552876] CPU 7: hi: 0, btch: 1 usd: 0 [37689.552882] CPU 8: hi: 0, btch: 1 usd: 0 [37689.552887] CPU 9: hi: 0, btch: 1 usd: 0 [37689.552891] CPU 10: hi: 0, btch: 1 usd: 0 [37689.552896] CPU 11: hi: 0, btch: 1 usd: 0 [37689.552902] CPU 12: hi: 0, btch: 1 usd: 0 [37689.552907] CPU 13: hi: 0, btch: 1 usd: 0 [37689.552912] CPU 14: hi: 0, btch: 1 usd: 0 [37689.552916] CPU 15: hi: 0, btch: 1 usd: 0 [37689.552921] CPU 16: hi: 0, btch: 1 usd: 0 [37689.552926] CPU 17: hi: 0, btch: 1 usd: 0 [37689.552930] CPU 18: hi: 0, btch: 1 usd: 0 [37689.552934] CPU 19: hi: 0, btch: 1 usd: 0 [37689.552939] CPU 20: hi: 0, btch: 1 usd: 0 [37689.552943] CPU 21: hi: 0, btch: 1 usd: 0 [37689.552948] CPU 22: hi: 0, btch: 1 usd: 0 [37689.552953] CPU 23: hi: 0, btch: 1 usd: 0 [37689.552958] CPU 24: hi: 0, btch: 1 usd: 0 [37689.552963] CPU 25: hi: 0, btch: 1 usd: 0 [37689.552968] CPU 26: hi: 0, btch: 1 usd: 0 [37689.552973] CPU 27: hi: 0, btch: 1 usd: 0 [37689.552977] CPU 28: hi: 0, btch: 1 usd: 0 [37689.552982] CPU 29: hi: 0, btch: 1 usd: 0 [37689.552987] CPU 30: hi: 0, btch: 1 usd: 0 [37689.552992] CPU 31: hi: 0, btch: 1 usd: 0 [37689.552996] Node 0 DMA32 per-cpu: [37689.553002] CPU 0: hi: 186, btch: 31 usd: 28 [37689.553007] CPU 1: hi: 186, btch: 31 usd: 65 [37689.553012] CPU 2: hi: 186, btch: 31 usd: 170 [37689.553017] CPU 3: hi: 186, btch: 31 usd: 3 [37689.553022] CPU 4: hi: 186, btch: 31 usd: 22 [37689.553028] CPU 5: hi: 186, btch: 31 usd: 182 [37689.553033] CPU 6: hi: 186, btch: 31 usd: 161 [37689.553038] CPU 7: hi: 186, btch: 31 usd: 171 [37689.553043] CPU 8: hi: 186, btch: 31 usd: 78 [37689.553048] CPU 9: hi: 186, btch: 31 usd: 128 [37689.553053] CPU 10: hi: 186, btch: 31 usd: 158 [37689.553058] CPU 11: hi: 186, btch: 31 usd: 67 [37689.553063] CPU 12: hi: 186, btch: 31 usd: 151 [37689.553068] CPU 13: hi: 186, btch: 31 usd: 7 [37689.553074] CPU 14: hi: 186, btch: 31 usd: 126 [37689.553079] CPU 15: hi: 186, btch: 31 usd: 173 [37689.553084] CPU 16: hi: 186, btch: 31 usd: 171 [37689.553089] CPU 17: hi: 186, btch: 31 usd: 165 [37689.553094] CPU 18: hi: 186, btch: 31 usd: 146 [37689.553099] CPU 19: hi: 186, btch: 31 usd: 164 [37689.553104] CPU 20: hi: 186, btch: 31 usd: 183 [37689.553108] CPU 21: hi: 186, btch: 31 usd: 185 [37689.553113] CPU 22: hi: 186, btch: 31 usd: 171 [37689.553117] CPU 23: hi: 186, btch: 31 usd: 159 [37689.553122] CPU 24: hi: 186, btch: 31 usd: 29 [37689.553126] CPU 25: hi: 186, btch: 31 usd: 128 [37689.553131] CPU 26: hi: 186, btch: 31 usd: 115 [37689.553136] CPU 27: hi: 186, btch: 31 usd: 171 [37689.553140] CPU 28: hi: 186, btch: 31 usd: 127 [37689.553145] CPU 29: hi: 186, btch: 31 usd: 155 [37689.553151] CPU 30: hi: 186, btch: 31 usd: 90 [37689.553156] CPU 31: hi: 186, btch: 31 usd: 150 [37689.553159] Node 0 Normal per-cpu: [37689.553165] CPU 0: hi: 186, btch: 31 usd: 23 [37689.553170] CPU 1: hi: 186, btch: 31 usd: 120 [37689.553175] CPU 2: hi: 186, btch: 31 usd: 174 [37689.553179] CPU 3: hi: 186, btch: 31 usd: 160 [37689.553184] CPU 4: hi: 186, btch: 31 usd: 159 [37689.553189] CPU 5: hi: 186, btch: 31 usd: 182 [37689.553194] CPU 6: hi: 186, btch: 31 usd: 198 [37689.553199] CPU 7: hi: 186, btch: 31 usd: 162 [37689.553204] CPU 8: hi: 186, btch: 31 usd: 0 [37689.553209] CPU 9: hi: 186, btch: 31 usd: 159 [37689.553214] CPU 10: hi: 186, btch: 31 usd: 160 [37689.553219] CPU 11: hi: 186, btch: 31 usd: 109 [37689.553224] CPU 12: hi: 186, btch: 31 usd: 172 [37689.553228] CPU 13: hi: 186, btch: 31 usd: 182 [37689.553233] CPU 14: hi: 186, btch: 31 usd: 159 [37689.553238] CPU 15: hi: 186, btch: 31 usd: 178 [37689.553242] CPU 16: hi: 186, btch: 31 usd: 174 [37689.553247] CPU 17: hi: 186, btch: 31 usd: 171 [37689.553252] CPU 18: hi: 186, btch: 31 usd: 166 [37689.553257] CPU 19: hi: 186, btch: 31 usd: 178 [37689.553262] CPU 20: hi: 186, btch: 31 usd: 177 [37689.553266] CPU 21: hi: 186, btch: 31 usd: 165 [37689.553271] CPU 22: hi: 186, btch: 31 usd: 162 [37689.553276] CPU 23: hi: 186, btch: 31 usd: 176 [37689.553280] CPU 24: hi: 186, btch: 31 usd: 0 [37689.553285] CPU 25: hi: 186, btch: 31 usd: 158 [37689.553290] CPU 26: hi: 186, btch: 31 usd: 98 [37689.553294] CPU 27: hi: 186, btch: 31 usd: 155 [37689.553299] CPU 28: hi: 186, btch: 31 usd: 1 [37689.553304] CPU 29: hi: 186, btch: 31 usd: 164 [37689.553308] CPU 30: hi: 186, btch: 31 usd: 173 [37689.553313] CPU 31: hi: 186, btch: 31 usd: 165 [37689.553317] Node 1 Normal per-cpu: [37689.553323] CPU 0: hi: 186, btch: 31 usd: 27 [37689.553327] CPU 1: hi: 186, btch: 31 usd: 29 [37689.553332] CPU 2: hi: 186, btch: 31 usd: 166 [37689.553337] CPU 3: hi: 186, btch: 31 usd: 159 [37689.553342] CPU 4: hi: 186, btch: 31 usd: 159 [37689.553347] CPU 5: hi: 186, btch: 31 usd: 155 [37689.553352] CPU 6: hi: 186, btch: 31 usd: 158 [37689.553357] CPU 7: hi: 186, btch: 31 usd: 175 [37689.553362] CPU 8: hi: 186, btch: 31 usd: 189 [37689.553367] CPU 9: hi: 186, btch: 31 usd: 164 [37689.553372] CPU 10: hi: 186, btch: 31 usd: 182 [37689.553377] CPU 11: hi: 186, btch: 31 usd: 171 [37689.553382] CPU 12: hi: 186, btch: 31 usd: 158 [37689.553387] CPU 13: hi: 186, btch: 31 usd: 177 [37689.553392] CPU 14: hi: 186, btch: 31 usd: 168 [37689.553397] CPU 15: hi: 186, btch: 31 usd: 166 [37689.553402] CPU 16: hi: 186, btch: 31 usd: 168 [37689.553406] CPU 17: hi: 186, btch: 31 usd: 163 [37689.553411] CPU 18: hi: 186, btch: 31 usd: 170 [37689.553416] CPU 19: hi: 186, btch: 31 usd: 166 [37689.553421] CPU 20: hi: 186, btch: 31 usd: 155 [37689.553426] CPU 21: hi: 186, btch: 31 usd: 183 [37689.553431] CPU 22: hi: 186, btch: 31 usd: 164 [37689.553436] CPU 23: hi: 186, btch: 31 usd: 161 [37689.553442] CPU 24: hi: 186, btch: 31 usd: 164 [37689.553447] CPU 25: hi: 186, btch: 31 usd: 176 [37689.553452] CPU 26: hi: 186, btch: 31 usd: 161 [37689.553456] CPU 27: hi: 186, btch: 31 usd: 164 [37689.553461] CPU 28: hi: 186, btch: 31 usd: 166 [37689.553467] CPU 29: hi: 186, btch: 31 usd: 160 [37689.553472] CPU 30: hi: 186, btch: 31 usd: 183 [37689.553477] CPU 31: hi: 186, btch: 31 usd: 157 [37689.553489] active_anon:175677 inactive_anon:26136 isolated_anon:0 [37689.553492] active_file:19617 inactive_file:14815845 isolated_file:32 [37689.553494] unevictable:2648 dirty:718 writeback:640 unstable:63145 [37689.553497] free:64104 slab_reclaimable:51903 slab_unreclaimable:869385 [37689.553499] mapped:12207 shmem:301 pagetables:2824 bounce:0 [37689.553504] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [37689.553525] lowmem_reserve[]: 0 2959 32249 32249 [37689.553533] Node 0 DMA32 free:122824kB min:4128kB low:5160kB high:6192kB active_anon:36kB inactive_anon:40964kB active_file:140kB inactive_file:1415880kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:436kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:39416kB slab_unreclaimable:842504kB kernel_stack:40kB pagetables:0kB unstable:13292kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37689.553555] lowmem_reserve[]: 0 0 29290 29290 [37689.553561] Node 0 Normal free:54316kB min:40864kB low:51080kB high:61296kB active_anon:502000kB inactive_anon:32660kB active_file:46596kB inactive_file:26735724kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:1852kB writeback:692kB mapped:30812kB shmem:300kB slab_reclaimable:108608kB slab_unreclaimable:1514700kB kernel_stack:2328kB pagetables:5736kB unstable:162208kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [37689.553586] lowmem_reserve[]: 0 0 0 0 [37689.553593] Node 1 Normal free:64548kB min:45092kB low:56364kB high:67636kB active_anon:200672kB inactive_anon:30920kB active_file:31732kB inactive_file:31111392kB unevictable:2064kB isolated(anon):0kB isolated(file):128kB present:33095680kB mlocked:2064kB dirty:584kB writeback:1868kB mapped:18016kB shmem:904kB slab_reclaimable:59588kB slab_unreclaimable:1120336kB kernel_stack:3072kB pagetables:5560kB unstable:77080kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37689.553619] lowmem_reserve[]: 0 0 0 0 [37689.553626] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [37689.553647] Node 0 DMA32: 30642*4kB 28*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 122824kB [37689.553666] Node 0 Normal: 13132*4kB 290*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54848kB [37689.553685] Node 1 Normal: 14229*4kB 729*8kB 144*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65052kB [37689.553705] 14835768 total pagecache pages [37689.553708] 0 pages in swap cache [37689.553713] Swap cache stats: add 0, delete 0, find 0/0 [37689.553717] Free swap = 8388604kB [37689.553720] Total swap = 8388604kB [37689.878676] 16777200 pages RAM [37689.878680] 393967 pages reserved [37689.878683] 3233209 pages shared [37689.878686] 13132441 pages non-shared [37689.878732] LNetError: 9234:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [37689.878744] LNetError: 9234:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315259042 total bytes allocated by lnet [37689.878756] LustreError: 9234:0:(niobuf.c:376:ptlrpc_register_bulk()) export-OST0009-osc-ffff88080dd47c00: LNetMDAttach failed x1456758787637276/0: rc = -12 [37689.878831] Lustre: 9234:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff8801ebcc3c00 x1456758787637276/t0(0) o4->export-OST0009-osc-ffff88080dd47c00@172.18.4.8@o2ib:6/4 lens 488/448 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [37689.878885] Lustre: export-OST0009-osc-ffff88080dd47c00: Connection to export-OST0009 (at 172.18.4.8@o2ib) was lost; in progress operations using this service will wait for recovery to complete [37689.904223] LustreError: 11-0: export-OST0009-osc-ffff88080dd47c00: Communicating with 172.18.4.8@o2ib, operation ost_connect failed with -16. [37714.868261] Lustre: export-OST0009-osc-ffff88080dd47c00: Connection restored to export-OST0009 (at 172.18.4.8@o2ib) [37724.346443] The following is only an harmless informational message. [37724.346451] Unless you get a _continuous_flood_ of these messages it means [37724.346453] everything is working fine. Allocations from irqs cannot be [37724.346456] perfectly reliable and the kernel is designed to handle that. [37724.346461] ptlrpcd_12: page allocation failure: order:1, mode:0x40 [37724.346467] Pid: 9227, comm: ptlrpcd_12 Tainted: P NX 3.0.51-0.7.9-default #1 [37724.346471] Call Trace: [37724.346498] [] dump_trace+0x75/0x300 [37724.346511] [] dump_stack+0x69/0x6f [37724.346522] [] warn_alloc_failed+0x102/0x1a0 [37724.346531] [] __alloc_pages_slowpath+0x541/0x7f0 [37724.346539] [] __alloc_pages_nodemask+0x1e9/0x200 [37724.346548] [] kmem_getpages+0x56/0x170 [37724.346556] [] fallback_alloc+0x19b/0x270 [37724.346563] [] __kmalloc+0x285/0x330 [37724.346592] [] LNetMDAttach+0x163/0x5b0 [lnet] [37724.346700] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [37724.346781] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [37724.346858] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [37724.346931] [] ptlrpc_check_set+0x408/0x1ae0 [ptlrpc] [37724.347015] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [37724.347118] [] ptlrpcd+0x32b/0x410 [ptlrpc] [37724.347177] [] kthread+0x96/0xa0 [37724.347187] [] kernel_thread_helper+0x4/0x10 [37724.347193] Mem-Info: [37724.347196] Node 0 DMA per-cpu: [37724.347200] CPU 0: hi: 0, btch: 1 usd: 0 [37724.347203] CPU 1: hi: 0, btch: 1 usd: 0 [37724.347206] CPU 2: hi: 0, btch: 1 usd: 0 [37724.347209] CPU 3: hi: 0, btch: 1 usd: 0 [37724.347212] CPU 4: hi: 0, btch: 1 usd: 0 [37724.347215] CPU 5: hi: 0, btch: 1 usd: 0 [37724.347218] CPU 6: hi: 0, btch: 1 usd: 0 [37724.347221] CPU 7: hi: 0, btch: 1 usd: 0 [37724.347224] CPU 8: hi: 0, btch: 1 usd: 0 [37724.347226] CPU 9: hi: 0, btch: 1 usd: 0 [37724.347229] CPU 10: hi: 0, btch: 1 usd: 0 [37724.347232] CPU 11: hi: 0, btch: 1 usd: 0 [37724.347235] CPU 12: hi: 0, btch: 1 usd: 0 [37724.347238] CPU 13: hi: 0, btch: 1 usd: 0 [37724.347241] CPU 14: hi: 0, btch: 1 usd: 0 [37724.347244] CPU 15: hi: 0, btch: 1 usd: 0 [37724.347247] CPU 16: hi: 0, btch: 1 usd: 0 [37724.347250] CPU 17: hi: 0, btch: 1 usd: 0 [37724.347252] CPU 18: hi: 0, btch: 1 usd: 0 [37724.347255] CPU 19: hi: 0, btch: 1 usd: 0 [37724.347258] CPU 20: hi: 0, btch: 1 usd: 0 [37724.347261] CPU 21: hi: 0, btch: 1 usd: 0 [37724.347264] CPU 22: hi: 0, btch: 1 usd: 0 [37724.347267] CPU 23: hi: 0, btch: 1 usd: 0 [37724.347270] CPU 24: hi: 0, btch: 1 usd: 0 [37724.347273] CPU 25: hi: 0, btch: 1 usd: 0 [37724.347276] CPU 26: hi: 0, btch: 1 usd: 0 [37724.347279] CPU 27: hi: 0, btch: 1 usd: 0 [37724.347281] CPU 28: hi: 0, btch: 1 usd: 0 [37724.347284] CPU 29: hi: 0, btch: 1 usd: 0 [37724.347287] CPU 30: hi: 0, btch: 1 usd: 0 [37724.347290] CPU 31: hi: 0, btch: 1 usd: 0 [37724.347293] Node 0 DMA32 per-cpu: [37724.347296] CPU 0: hi: 186, btch: 31 usd: 42 [37724.347300] CPU 1: hi: 186, btch: 31 usd: 39 [37724.347303] CPU 2: hi: 186, btch: 31 usd: 44 [37724.347306] CPU 3: hi: 186, btch: 31 usd: 31 [37724.347309] CPU 4: hi: 186, btch: 31 usd: 158 [37724.347312] CPU 5: hi: 186, btch: 31 usd: 184 [37724.347315] CPU 6: hi: 186, btch: 31 usd: 155 [37724.347318] CPU 7: hi: 186, btch: 31 usd: 185 [37724.347321] CPU 8: hi: 186, btch: 31 usd: 78 [37724.347324] CPU 9: hi: 186, btch: 31 usd: 0 [37724.347327] CPU 10: hi: 186, btch: 31 usd: 0 [37724.347330] CPU 11: hi: 186, btch: 31 usd: 119 [37724.347333] CPU 12: hi: 186, btch: 31 usd: 112 [37724.347336] CPU 13: hi: 186, btch: 31 usd: 142 [37724.347338] CPU 14: hi: 186, btch: 31 usd: 153 [37724.347341] CPU 15: hi: 186, btch: 31 usd: 87 [37724.347344] CPU 16: hi: 186, btch: 31 usd: 170 [37724.347347] CPU 17: hi: 186, btch: 31 usd: 175 [37724.347350] CPU 18: hi: 186, btch: 31 usd: 118 [37724.347353] CPU 19: hi: 186, btch: 31 usd: 172 [37724.347356] CPU 20: hi: 186, btch: 31 usd: 172 [37724.347359] CPU 21: hi: 186, btch: 31 usd: 158 [37724.347362] CPU 22: hi: 186, btch: 31 usd: 171 [37724.347365] CPU 23: hi: 186, btch: 31 usd: 159 [37724.347368] CPU 24: hi: 186, btch: 31 usd: 30 [37724.347371] CPU 25: hi: 186, btch: 31 usd: 128 [37724.347374] CPU 26: hi: 186, btch: 31 usd: 84 [37724.347377] CPU 27: hi: 186, btch: 31 usd: 140 [37724.347380] CPU 28: hi: 186, btch: 31 usd: 127 [37724.347383] CPU 29: hi: 186, btch: 31 usd: 93 [37724.347386] CPU 30: hi: 186, btch: 31 usd: 0 [37724.347389] CPU 31: hi: 186, btch: 31 usd: 181 [37724.347391] Node 0 Normal per-cpu: [37724.347394] CPU 0: hi: 186, btch: 31 usd: 98 [37724.347397] CPU 1: hi: 186, btch: 31 usd: 52 [37724.347400] CPU 2: hi: 186, btch: 31 usd: 172 [37724.347403] CPU 3: hi: 186, btch: 31 usd: 170 [37724.347406] CPU 4: hi: 186, btch: 31 usd: 155 [37724.347409] CPU 5: hi: 186, btch: 31 usd: 160 [37724.347412] CPU 6: hi: 186, btch: 31 usd: 156 [37724.347415] CPU 7: hi: 186, btch: 31 usd: 183 [37724.347418] CPU 8: hi: 186, btch: 31 usd: 0 [37724.347421] CPU 9: hi: 186, btch: 31 usd: 2 [37724.347424] CPU 10: hi: 186, btch: 31 usd: 170 [37724.347427] CPU 11: hi: 186, btch: 31 usd: 170 [37724.347429] CPU 12: hi: 186, btch: 31 usd: 174 [37724.347432] CPU 13: hi: 186, btch: 31 usd: 171 [37724.347435] CPU 14: hi: 186, btch: 31 usd: 162 [37724.347438] CPU 15: hi: 186, btch: 31 usd: 135 [37724.347441] CPU 16: hi: 186, btch: 31 usd: 158 [37724.347444] CPU 17: hi: 186, btch: 31 usd: 57 [37724.347447] CPU 18: hi: 186, btch: 31 usd: 166 [37724.347449] CPU 19: hi: 186, btch: 31 usd: 166 [37724.347452] CPU 20: hi: 186, btch: 31 usd: 160 [37724.347455] CPU 21: hi: 186, btch: 31 usd: 175 [37724.347458] CPU 22: hi: 186, btch: 31 usd: 180 [37724.347461] CPU 23: hi: 186, btch: 31 usd: 206 [37724.347464] CPU 24: hi: 186, btch: 31 usd: 32 [37724.347466] CPU 25: hi: 186, btch: 31 usd: 126 [37724.347469] CPU 26: hi: 186, btch: 31 usd: 67 [37724.347472] CPU 27: hi: 186, btch: 31 usd: 124 [37724.347475] CPU 28: hi: 186, btch: 31 usd: 1 [37724.347478] CPU 29: hi: 186, btch: 31 usd: 72 [37724.347481] CPU 30: hi: 186, btch: 31 usd: 80 [37724.347484] CPU 31: hi: 186, btch: 31 usd: 175 [37724.347487] Node 1 Normal per-cpu: [37724.347490] CPU 0: hi: 186, btch: 31 usd: 0 [37724.347493] CPU 1: hi: 186, btch: 31 usd: 30 [37724.347496] CPU 2: hi: 186, btch: 31 usd: 184 [37724.347499] CPU 3: hi: 186, btch: 31 usd: 175 [37724.347502] CPU 4: hi: 186, btch: 31 usd: 168 [37724.347505] CPU 5: hi: 186, btch: 31 usd: 45 [37724.347508] CPU 6: hi: 186, btch: 31 usd: 181 [37724.347511] CPU 7: hi: 186, btch: 31 usd: 163 [37724.347514] CPU 8: hi: 186, btch: 31 usd: 158 [37724.347517] CPU 9: hi: 186, btch: 31 usd: 173 [37724.347520] CPU 10: hi: 186, btch: 31 usd: 176 [37724.347523] CPU 11: hi: 186, btch: 31 usd: 176 [37724.347526] CPU 12: hi: 186, btch: 31 usd: 205 [37724.347528] CPU 13: hi: 186, btch: 31 usd: 168 [37724.347531] CPU 14: hi: 186, btch: 31 usd: 162 [37724.347534] CPU 15: hi: 186, btch: 31 usd: 184 [37724.347537] CPU 16: hi: 186, btch: 31 usd: 76 [37724.347540] CPU 17: hi: 186, btch: 31 usd: 89 [37724.347544] CPU 18: hi: 186, btch: 31 usd: 25 [37724.347547] CPU 19: hi: 186, btch: 31 usd: 105 [37724.347550] CPU 20: hi: 186, btch: 31 usd: 184 [37724.347553] CPU 21: hi: 186, btch: 31 usd: 142 [37724.347556] CPU 22: hi: 186, btch: 31 usd: 179 [37724.347559] CPU 23: hi: 186, btch: 31 usd: 130 [37724.347562] CPU 24: hi: 186, btch: 31 usd: 157 [37724.347565] CPU 25: hi: 186, btch: 31 usd: 172 [37724.347567] CPU 26: hi: 186, btch: 31 usd: 163 [37724.347570] CPU 27: hi: 186, btch: 31 usd: 194 [37724.347573] CPU 28: hi: 186, btch: 31 usd: 165 [37724.347576] CPU 29: hi: 186, btch: 31 usd: 158 [37724.347579] CPU 30: hi: 186, btch: 31 usd: 170 [37724.347582] CPU 31: hi: 186, btch: 31 usd: 174 [37724.347590] active_anon:175648 inactive_anon:26136 isolated_anon:0 [37724.347592] active_file:20189 inactive_file:14822229 isolated_file:32 [37724.347594] unevictable:2648 dirty:735 writeback:324 unstable:93849 [37724.347595] free:71962 slab_reclaimable:51869 slab_unreclaimable:855818 [37724.347597] mapped:12206 shmem:301 pagetables:2794 bounce:0 [37724.347600] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [37724.347616] lowmem_reserve[]: 0 2959 32249 32249 [37724.347622] Node 0 DMA32 free:123396kB min:4128kB low:5160kB high:6192kB active_anon:36kB inactive_anon:40964kB active_file:172kB inactive_file:1423456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:0kB writeback:488kB mapped:0kB shmem:0kB slab_reclaimable:39416kB slab_unreclaimable:836172kB kernel_stack:40kB pagetables:0kB unstable:10636kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37724.347639] lowmem_reserve[]: 0 0 29290 29290 [37724.347644] Node 0 Normal free:68036kB min:40864kB low:51080kB high:61296kB active_anon:501948kB inactive_anon:32660kB active_file:48632kB inactive_file:26767840kB unevictable:8528kB isolated(anon):0kB isolated(file):128kB present:29992960kB mlocked:2396kB dirty:1552kB writeback:572kB mapped:30808kB shmem:300kB slab_reclaimable:108608kB slab_unreclaimable:1467952kB kernel_stack:2328kB pagetables:5624kB unstable:275124kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [37724.347661] lowmem_reserve[]: 0 0 0 0 [37724.347666] Node 1 Normal free:81316kB min:45092kB low:56364kB high:67636kB active_anon:200608kB inactive_anon:30920kB active_file:31952kB inactive_file:31097552kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:1472kB writeback:236kB mapped:18016kB shmem:904kB slab_reclaimable:59452kB slab_unreclaimable:1119148kB kernel_stack:3072kB pagetables:5552kB unstable:89636kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no [37724.347683] lowmem_reserve[]: 0 0 0 0 [37724.347687] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [37724.347700] Node 0 DMA32: 30709*4kB 68*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 123380kB [37724.347712] Node 0 Normal: 13489*4kB 1705*8kB 136*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 69772kB [37724.347725] Node 1 Normal: 16551*4kB 1256*8kB 218*16kB 48*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 81276kB [37724.347738] 14843149 total pagecache pages [37724.347740] 0 pages in swap cache [37724.347743] Swap cache stats: add 0, delete 0, find 0/0 [37724.347746] Free swap = 8388604kB [37724.347748] Total swap = 8388604kB [37724.657963] 16777200 pages RAM [37724.657967] 393967 pages reserved [37724.657970] 3218571 pages shared [37724.657971] 13179579 pages non-shared [37724.658015] LNetError: 9227:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [37724.658034] LNetError: 9227:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315258594 total bytes allocated by lnet [37724.658059] LustreError: 9227:0:(niobuf.c:376:ptlrpc_register_bulk()) export-OST0009-osc-ffff88080dd47c00: LNetMDAttach failed x1456758787645116/0: rc = -12 [37724.658110] Lustre: 9227:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880e40438800 x1456758787645116/t0(0) o4->export-OST0009-osc-ffff88080dd47c00@172.18.4.8@o2ib:6/4 lens 488/448 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [37724.658144] Lustre: export-OST0009-osc-ffff88080dd47c00: Connection to export-OST0009 (at 172.18.4.8@o2ib) was lost; in progress operations using this service will wait for recovery to complete [37724.672979] LustreError: 11-0: export-OST0009-osc-ffff88080dd47c00: Communicating with 172.18.4.8@o2ib, operation ost_connect failed with -16. [37749.657735] Lustre: export-OST0009-osc-ffff88080dd47c00: Connection restored to export-OST0009 (at 172.18.4.8@o2ib) [38504.391781] The following is only an harmless informational message. [38504.391785] Unless you get a _continuous_flood_ of these messages it means [38504.391787] everything is working fine. Allocations from irqs cannot be [38504.391789] perfectly reliable and the kernel is designed to handle that. [38504.391793] ptlrpcd_9: page allocation failure: order:1, mode:0x40 [38504.391798] Pid: 9224, comm: ptlrpcd_9 Tainted: P NX 3.0.51-0.7.9-default #1 [38504.391801] Call Trace: [38504.391843] [] dump_trace+0x75/0x300 [38504.391863] [] dump_stack+0x69/0x6f [38504.391874] [] warn_alloc_failed+0x102/0x1a0 [38504.391883] [] __alloc_pages_slowpath+0x541/0x7f0 [38504.391891] [] __alloc_pages_nodemask+0x1e9/0x200 [38504.391901] [] kmem_getpages+0x56/0x170 [38504.391909] [] fallback_alloc+0x19b/0x270 [38504.391916] [] __kmalloc+0x285/0x330 [38504.391993] [] ptlrpc_new_bulk+0x45/0x2a0 [ptlrpc] [38504.392097] [] ptlrpc_prep_bulk_imp+0x62/0x1b0 [ptlrpc] [38504.392161] [] osc_brw_prep_request+0x1bf/0xf20 [osc] [38504.392184] [] osc_build_rpc+0x7ba/0x1380 [osc] [38504.392210] [] osc_send_write_rpc+0x1e5/0x470 [osc] [38504.392247] [] osc_check_rpcs+0x5b4/0x710 [osc] [38504.392284] [] osc_io_unplug0+0x35b/0x4c0 [osc] [38504.392318] [] brw_queue_work+0x30/0xd0 [osc] [38504.392358] [] ptlrpc_check_set+0x522/0x1ae0 [ptlrpc] [38504.392426] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [38504.392516] [] ptlrpcd+0x27b/0x410 [ptlrpc] [38504.392597] [] kthread+0x96/0xa0 [38504.392607] [] kernel_thread_helper+0x4/0x10 [38504.392612] Mem-Info: [38504.392615] Node 0 DMA per-cpu: [38504.392618] CPU 0: hi: 0, btch: 1 usd: 0 [38504.392621] CPU 1: hi: 0, btch: 1 usd: 0 [38504.392623] CPU 2: hi: 0, btch: 1 usd: 0 [38504.392626] CPU 3: hi: 0, btch: 1 usd: 0 [38504.392628] CPU 4: hi: 0, btch: 1 usd: 0 [38504.392631] CPU 5: hi: 0, btch: 1 usd: 0 [38504.392634] CPU 6: hi: 0, btch: 1 usd: 0 [38504.392636] CPU 7: hi: 0, btch: 1 usd: 0 [38504.392639] CPU 8: hi: 0, btch: 1 usd: 0 [38504.392641] CPU 9: hi: 0, btch: 1 usd: 0 [38504.392644] CPU 10: hi: 0, btch: 1 usd: 0 [38504.392646] CPU 11: hi: 0, btch: 1 usd: 0 [38504.392649] CPU 12: hi: 0, btch: 1 usd: 0 [38504.392651] CPU 13: hi: 0, btch: 1 usd: 0 [38504.392654] CPU 14: hi: 0, btch: 1 usd: 0 [38504.392656] CPU 15: hi: 0, btch: 1 usd: 0 [38504.392659] CPU 16: hi: 0, btch: 1 usd: 0 [38504.392662] CPU 17: hi: 0, btch: 1 usd: 0 [38504.392664] CPU 18: hi: 0, btch: 1 usd: 0 [38504.392667] CPU 19: hi: 0, btch: 1 usd: 0 [38504.392669] CPU 20: hi: 0, btch: 1 usd: 0 [38504.392672] CPU 21: hi: 0, btch: 1 usd: 0 [38504.392675] CPU 22: hi: 0, btch: 1 usd: 0 [38504.392677] CPU 23: hi: 0, btch: 1 usd: 0 [38504.392679] CPU 24: hi: 0, btch: 1 usd: 0 [38504.392682] CPU 25: hi: 0, btch: 1 usd: 0 [38504.392684] CPU 26: hi: 0, btch: 1 usd: 0 [38504.392687] CPU 27: hi: 0, btch: 1 usd: 0 [38504.392690] CPU 28: hi: 0, btch: 1 usd: 0 [38504.392692] CPU 29: hi: 0, btch: 1 usd: 0 [38504.392695] CPU 30: hi: 0, btch: 1 usd: 0 [38504.392698] CPU 31: hi: 0, btch: 1 usd: 0 [38504.392700] Node 0 DMA32 per-cpu: [38504.392703] CPU 0: hi: 186, btch: 31 usd: 162 [38504.392706] CPU 1: hi: 186, btch: 31 usd: 184 [38504.392708] CPU 2: hi: 186, btch: 31 usd: 179 [38504.392711] CPU 3: hi: 186, btch: 31 usd: 161 [38504.392713] CPU 4: hi: 186, btch: 31 usd: 1 [38504.392716] CPU 5: hi: 186, btch: 31 usd: 13 [38504.392718] CPU 6: hi: 186, btch: 31 usd: 167 [38504.392721] CPU 7: hi: 186, btch: 31 usd: 38 [38504.392723] CPU 8: hi: 186, btch: 31 usd: 60 [38504.392726] CPU 9: hi: 186, btch: 31 usd: 75 [38504.392728] CPU 10: hi: 186, btch: 31 usd: 25 [38504.392731] CPU 11: hi: 186, btch: 31 usd: 184 [38504.392733] CPU 12: hi: 186, btch: 31 usd: 51 [38504.392735] CPU 13: hi: 186, btch: 31 usd: 64 [38504.392738] CPU 14: hi: 186, btch: 31 usd: 95 [38504.392740] CPU 15: hi: 186, btch: 31 usd: 24 [38504.392742] CPU 16: hi: 186, btch: 31 usd: 174 [38504.392744] CPU 17: hi: 186, btch: 31 usd: 184 [38504.392747] CPU 18: hi: 186, btch: 31 usd: 160 [38504.392749] CPU 19: hi: 186, btch: 31 usd: 169 [38504.392751] CPU 20: hi: 186, btch: 31 usd: 175 [38504.392753] CPU 21: hi: 186, btch: 31 usd: 166 [38504.392756] CPU 22: hi: 186, btch: 31 usd: 181 [38504.392758] CPU 23: hi: 186, btch: 31 usd: 159 [38504.392760] CPU 24: hi: 186, btch: 31 usd: 42 [38504.392762] CPU 25: hi: 186, btch: 31 usd: 172 [38504.392764] CPU 26: hi: 186, btch: 31 usd: 160 [38504.392766] CPU 27: hi: 186, btch: 31 usd: 185 [38504.392767] CPU 28: hi: 186, btch: 31 usd: 168 [38504.392769] CPU 29: hi: 186, btch: 31 usd: 164 [38504.392771] CPU 30: hi: 186, btch: 31 usd: 126 [38504.392773] CPU 31: hi: 186, btch: 31 usd: 108 [38504.392775] Node 0 Normal per-cpu: [38504.392778] CPU 0: hi: 186, btch: 31 usd: 183 [38504.392780] CPU 1: hi: 186, btch: 31 usd: 169 [38504.392783] CPU 2: hi: 186, btch: 31 usd: 171 [38504.392785] CPU 3: hi: 186, btch: 31 usd: 180 [38504.392788] CPU 4: hi: 186, btch: 31 usd: 8 [38504.392789] CPU 5: hi: 186, btch: 31 usd: 19 [38504.392791] CPU 6: hi: 186, btch: 31 usd: 161 [38504.392792] CPU 7: hi: 186, btch: 31 usd: 97 [38504.392794] CPU 8: hi: 186, btch: 31 usd: 173 [38504.392796] CPU 9: hi: 186, btch: 31 usd: 149 [38504.392797] CPU 10: hi: 186, btch: 31 usd: 104 [38504.392799] CPU 11: hi: 186, btch: 31 usd: 180 [38504.392800] CPU 12: hi: 186, btch: 31 usd: 180 [38504.392815] CPU 13: hi: 186, btch: 31 usd: 165 [38504.392818] CPU 14: hi: 186, btch: 31 usd: 161 [38504.392821] CPU 15: hi: 186, btch: 31 usd: 6 [38504.392824] CPU 16: hi: 186, btch: 31 usd: 180 [38504.392827] CPU 17: hi: 186, btch: 31 usd: 180 [38504.392830] CPU 18: hi: 186, btch: 31 usd: 174 [38504.392833] CPU 19: hi: 186, btch: 31 usd: 181 [38504.392836] CPU 20: hi: 186, btch: 31 usd: 168 [38504.392839] CPU 21: hi: 186, btch: 31 usd: 174 [38504.392842] CPU 22: hi: 186, btch: 31 usd: 171 [38504.392845] CPU 23: hi: 186, btch: 31 usd: 98 [38504.392847] CPU 24: hi: 186, btch: 31 usd: 172 [38504.392850] CPU 25: hi: 186, btch: 31 usd: 168 [38504.392853] CPU 26: hi: 186, btch: 31 usd: 146 [38504.392856] CPU 27: hi: 186, btch: 31 usd: 159 [38504.392859] CPU 28: hi: 186, btch: 31 usd: 171 [38504.392862] CPU 29: hi: 186, btch: 31 usd: 162 [38504.392865] CPU 30: hi: 186, btch: 31 usd: 95 [38504.392868] CPU 31: hi: 186, btch: 31 usd: 98 [38504.392871] Node 1 Normal per-cpu: [38504.392874] CPU 0: hi: 186, btch: 31 usd: 181 [38504.392877] CPU 1: hi: 186, btch: 31 usd: 172 [38504.392880] CPU 2: hi: 186, btch: 31 usd: 165 [38504.392884] CPU 3: hi: 186, btch: 31 usd: 163 [38504.392887] CPU 4: hi: 186, btch: 31 usd: 5 [38504.392890] CPU 5: hi: 186, btch: 31 usd: 30 [38504.392893] CPU 6: hi: 186, btch: 31 usd: 170 [38504.392896] CPU 7: hi: 186, btch: 31 usd: 0 [38504.392898] CPU 8: hi: 186, btch: 31 usd: 180 [38504.392901] CPU 9: hi: 186, btch: 31 usd: 69 [38504.392904] CPU 10: hi: 186, btch: 31 usd: 40 [38504.392907] CPU 11: hi: 186, btch: 31 usd: 170 [38504.392910] CPU 12: hi: 186, btch: 31 usd: 77 [38504.392913] CPU 13: hi: 186, btch: 31 usd: 158 [38504.392916] CPU 14: hi: 186, btch: 31 usd: 185 [38504.392919] CPU 15: hi: 186, btch: 31 usd: 109 [38504.392922] CPU 16: hi: 186, btch: 31 usd: 134 [38504.392925] CPU 17: hi: 186, btch: 31 usd: 167 [38504.392928] CPU 18: hi: 186, btch: 31 usd: 184 [38504.392931] CPU 19: hi: 186, btch: 31 usd: 133 [38504.392934] CPU 20: hi: 186, btch: 31 usd: 171 [38504.392937] CPU 21: hi: 186, btch: 31 usd: 181 [38504.392940] CPU 22: hi: 186, btch: 31 usd: 165 [38504.392943] CPU 23: hi: 186, btch: 31 usd: 125 [38504.392946] CPU 24: hi: 186, btch: 31 usd: 212 [38504.392949] CPU 25: hi: 186, btch: 31 usd: 163 [38504.392952] CPU 26: hi: 186, btch: 31 usd: 49 [38504.392955] CPU 27: hi: 186, btch: 31 usd: 168 [38504.392958] CPU 28: hi: 186, btch: 31 usd: 170 [38504.392961] CPU 29: hi: 186, btch: 31 usd: 173 [38504.392964] CPU 30: hi: 186, btch: 31 usd: 183 [38504.392967] CPU 31: hi: 186, btch: 31 usd: 151 [38504.392974] active_anon:361397 inactive_anon:37615 isolated_anon:0 [38504.392976] active_file:24360 inactive_file:10988599 isolated_file:0 [38504.392978] unevictable:2648 dirty:3032 writeback:726 unstable:61631 [38504.392979] free:63865 slab_reclaimable:3124558 slab_unreclaimable:576198 [38504.392981] mapped:12245 shmem:301 pagetables:3315 bounce:0 [38504.392985] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [38504.393001] lowmem_reserve[]: 0 2959 32249 32249 [38504.393006] Node 0 DMA32 free:123352kB min:4128kB low:5160kB high:6192kB active_anon:7452kB inactive_anon:40964kB active_file:316kB inactive_file:1031860kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:896kB writeback:1876kB mapped:44kB shmem:0kB slab_reclaimable:895884kB slab_unreclaimable:295044kB kernel_stack:56kB pagetables:44kB unstable:12980kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [38504.393023] lowmem_reserve[]: 0 0 29290 29290 [38504.393028] Node 0 Normal free:54796kB min:40864kB low:51080kB high:61296kB active_anon:519072kB inactive_anon:52364kB active_file:41952kB inactive_file:17509148kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:1864kB writeback:516kB mapped:30920kB shmem:300kB slab_reclaimable:7695972kB slab_unreclaimable:1064248kB kernel_stack:2360kB pagetables:6100kB unstable:221756kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no [38504.393046] lowmem_reserve[]: 0 0 0 0 [38504.393050] Node 1 Normal free:61840kB min:45092kB low:56364kB high:67636kB active_anon:919064kB inactive_anon:57132kB active_file:55172kB inactive_file:25413388kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:9368kB writeback:512kB mapped:18016kB shmem:904kB slab_reclaimable:3906376kB slab_unreclaimable:945500kB kernel_stack:3112kB pagetables:7116kB unstable:11788kB bounce:0kB writeback_tmp:0kB pages_scanned:10 all_unreclaimable? no [38504.393072] lowmem_reserve[]: 0 0 0 0 [38504.393079] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [38504.393101] Node 0 DMA32: 30761*4kB 9*8kB 6*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 123212kB [38504.393111] Node 0 Normal: 13273*4kB 245*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 55084kB [38504.393121] Node 1 Normal: 12876*4kB 1356*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62368kB [38504.393131] 11013681 total pagecache pages [38504.393133] 0 pages in swap cache [38504.393135] Swap cache stats: add 0, delete 0, find 0/0 [38504.393137] Free swap = 8388604kB [38504.393139] Total swap = 8388604kB [38504.689267] 16777200 pages RAM [38504.689271] 393967 pages reserved [38504.689273] 1608587 pages shared [38504.689274] 14760186 pages non-shared [38504.689321] LustreError: 9224:0:(osc_request.c:2198:osc_build_rpc()) prep_req failed: -12 [38504.689975] LustreError: 9224:0:(osc_cache.c:2211:osc_check_rpcs()) Write request failed with -12 [42699.436201] The following is only an harmless informational message. [42699.436206] Unless you get a _continuous_flood_ of these messages it means [42699.436209] everything is working fine. Allocations from irqs cannot be [42699.436212] perfectly reliable and the kernel is designed to handle that. [42699.436216] find: page allocation failure: order:1, mode:0x40 [42699.436221] Pid: 98842, comm: find Tainted: P NX 3.0.51-0.7.9-default #1 [42699.436226] Call Trace: [42699.436255] [] dump_trace+0x75/0x300 [42699.436275] [] dump_stack+0x69/0x6f [42699.436286] [] warn_alloc_failed+0x102/0x1a0 [42699.436295] [] __alloc_pages_slowpath+0x541/0x7f0 [42699.436303] [] __alloc_pages_nodemask+0x1e9/0x200 [42699.436313] [] kmem_getpages+0x56/0x170 [42699.436320] [] fallback_alloc+0x19b/0x270 [42699.436327] [] __kmalloc+0x285/0x330 [42699.436356] [] LNetMDAttach+0x163/0x5b0 [lnet] [42699.436464] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [42699.436546] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [42699.436624] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [42699.436697] [] ptlrpc_set_wait+0x664/0x8f0 [ptlrpc] [42699.436770] [] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [42699.436812] [] mdc_readpage+0x169/0x630 [mdc] [42699.436834] [] lmv_readpage+0x2dd/0x710 [lmv] [42699.436875] [] ll_dir_filler+0x38c/0xb00 [lustre] [42699.436889] [] __read_cache_page+0x85/0xf0 [42699.436898] [] do_read_cache_page+0x34/0xd0 [42699.436906] [] read_cache_page+0x9/0x20 [42699.436934] [] ll_get_dir_page+0xe83/0x1260 [lustre] [42699.436966] [] ll_dir_read+0x94/0x380 [lustre] [42699.436997] [] ll_readdir+0x16c/0x360 [lustre] [42699.437012] [] vfs_readdir+0xc2/0xe0 [42699.437021] [] sys_getdents64+0x84/0xe0 [42699.437031] [] system_call_fastpath+0x16/0x1b [42699.437074] [<00007f76aed61f9a>] 0x7f76aed61f99 [42699.437077] Mem-Info: [42699.437080] Node 0 DMA per-cpu: [42699.437084] CPU 0: hi: 0, btch: 1 usd: 0 [42699.437087] CPU 1: hi: 0, btch: 1 usd: 0 [42699.437090] CPU 2: hi: 0, btch: 1 usd: 0 [42699.437093] CPU 3: hi: 0, btch: 1 usd: 0 [42699.437096] CPU 4: hi: 0, btch: 1 usd: 0 [42699.437099] CPU 5: hi: 0, btch: 1 usd: 0 [42699.437102] CPU 6: hi: 0, btch: 1 usd: 0 [42699.437105] CPU 7: hi: 0, btch: 1 usd: 0 [42699.437108] CPU 8: hi: 0, btch: 1 usd: 0 [42699.437110] CPU 9: hi: 0, btch: 1 usd: 0 [42699.437113] CPU 10: hi: 0, btch: 1 usd: 0 [42699.437116] CPU 11: hi: 0, btch: 1 usd: 0 [42699.437119] CPU 12: hi: 0, btch: 1 usd: 0 [42699.437122] CPU 13: hi: 0, btch: 1 usd: 0 [42699.437125] CPU 14: hi: 0, btch: 1 usd: 0 [42699.437128] CPU 15: hi: 0, btch: 1 usd: 0 [42699.437131] CPU 16: hi: 0, btch: 1 usd: 0 [42699.437134] CPU 17: hi: 0, btch: 1 usd: 0 [42699.437137] CPU 18: hi: 0, btch: 1 usd: 0 [42699.437140] CPU 19: hi: 0, btch: 1 usd: 0 [42699.437143] CPU 20: hi: 0, btch: 1 usd: 0 [42699.437145] CPU 21: hi: 0, btch: 1 usd: 0 [42699.437148] CPU 22: hi: 0, btch: 1 usd: 0 [42699.437151] CPU 23: hi: 0, btch: 1 usd: 0 [42699.437154] CPU 24: hi: 0, btch: 1 usd: 0 [42699.437157] CPU 25: hi: 0, btch: 1 usd: 0 [42699.437160] CPU 26: hi: 0, btch: 1 usd: 0 [42699.437163] CPU 27: hi: 0, btch: 1 usd: 0 [42699.437166] CPU 28: hi: 0, btch: 1 usd: 0 [42699.437168] CPU 29: hi: 0, btch: 1 usd: 0 [42699.437171] CPU 30: hi: 0, btch: 1 usd: 0 [42699.437174] CPU 31: hi: 0, btch: 1 usd: 0 [42699.437176] Node 0 DMA32 per-cpu: [42699.437180] CPU 0: hi: 186, btch: 31 usd: 176 [42699.437184] CPU 1: hi: 186, btch: 31 usd: 179 [42699.437187] CPU 2: hi: 186, btch: 31 usd: 30 [42699.437190] CPU 3: hi: 186, btch: 31 usd: 0 [42699.437193] CPU 4: hi: 186, btch: 31 usd: 30 [42699.437196] CPU 5: hi: 186, btch: 31 usd: 6 [42699.437199] CPU 6: hi: 186, btch: 31 usd: 26 [42699.437202] CPU 7: hi: 186, btch: 31 usd: 25 [42699.437205] CPU 8: hi: 186, btch: 31 usd: 0 [42699.437208] CPU 9: hi: 186, btch: 31 usd: 0 [42699.437210] CPU 10: hi: 186, btch: 31 usd: 0 [42699.437213] CPU 11: hi: 186, btch: 31 usd: 0 [42699.437216] CPU 12: hi: 186, btch: 31 usd: 135 [42699.437219] CPU 13: hi: 186, btch: 31 usd: 126 [42699.437222] CPU 14: hi: 186, btch: 31 usd: 0 [42699.437225] CPU 15: hi: 186, btch: 31 usd: 105 [42699.437228] CPU 16: hi: 186, btch: 31 usd: 184 [42699.437231] CPU 17: hi: 186, btch: 31 usd: 171 [42699.437234] CPU 18: hi: 186, btch: 31 usd: 91 [42699.437237] CPU 19: hi: 186, btch: 31 usd: 58 [42699.437240] CPU 20: hi: 186, btch: 31 usd: 60 [42699.437243] CPU 21: hi: 186, btch: 31 usd: 155 [42699.437246] CPU 22: hi: 186, btch: 31 usd: 46 [42699.437249] CPU 23: hi: 186, btch: 31 usd: 77 [42699.437252] CPU 24: hi: 186, btch: 31 usd: 0 [42699.437255] CPU 25: hi: 186, btch: 31 usd: 0 [42699.437258] CPU 26: hi: 186, btch: 31 usd: 0 [42699.437261] CPU 27: hi: 186, btch: 31 usd: 0 [42699.437264] CPU 28: hi: 186, btch: 31 usd: 0 [42699.437267] CPU 29: hi: 186, btch: 31 usd: 127 [42699.437270] CPU 30: hi: 186, btch: 31 usd: 53 [42699.437273] CPU 31: hi: 186, btch: 31 usd: 0 [42699.437275] Node 0 Normal per-cpu: [42699.437278] CPU 0: hi: 186, btch: 31 usd: 181 [42699.437281] CPU 1: hi: 186, btch: 31 usd: 155 [42699.437284] CPU 2: hi: 186, btch: 31 usd: 23 [42699.437287] CPU 3: hi: 186, btch: 31 usd: 157 [42699.437290] CPU 4: hi: 186, btch: 31 usd: 48 [42699.437293] CPU 5: hi: 186, btch: 31 usd: 36 [42699.437296] CPU 6: hi: 186, btch: 31 usd: 180 [42699.437299] CPU 7: hi: 186, btch: 31 usd: 175 [42699.437302] CPU 8: hi: 186, btch: 31 usd: 6 [42699.437305] CPU 9: hi: 186, btch: 31 usd: 13 [42699.437307] CPU 10: hi: 186, btch: 31 usd: 14 [42699.437310] CPU 11: hi: 186, btch: 31 usd: 0 [42699.437313] CPU 12: hi: 186, btch: 31 usd: 162 [42699.437316] CPU 13: hi: 186, btch: 31 usd: 144 [42699.437319] CPU 14: hi: 186, btch: 31 usd: 159 [42699.437322] CPU 15: hi: 186, btch: 31 usd: 132 [42699.437325] CPU 16: hi: 186, btch: 31 usd: 17 [42699.437328] CPU 17: hi: 186, btch: 31 usd: 184 [42699.437331] CPU 18: hi: 186, btch: 31 usd: 178 [42699.437333] CPU 19: hi: 186, btch: 31 usd: 133 [42699.437336] CPU 20: hi: 186, btch: 31 usd: 69 [42699.437339] CPU 21: hi: 186, btch: 31 usd: 165 [42699.437342] CPU 22: hi: 186, btch: 31 usd: 169 [42699.437345] CPU 23: hi: 186, btch: 31 usd: 22 [42699.437348] CPU 24: hi: 186, btch: 31 usd: 0 [42699.437351] CPU 25: hi: 186, btch: 31 usd: 185 [42699.437354] CPU 26: hi: 186, btch: 31 usd: 0 [42699.437357] CPU 27: hi: 186, btch: 31 usd: 50 [42699.437360] CPU 28: hi: 186, btch: 31 usd: 0 [42699.437362] CPU 29: hi: 186, btch: 31 usd: 62 [42699.437365] CPU 30: hi: 186, btch: 31 usd: 33 [42699.437368] CPU 31: hi: 186, btch: 31 usd: 6 [42699.437371] Node 1 Normal per-cpu: [42699.437374] CPU 0: hi: 186, btch: 31 usd: 167 [42699.437377] CPU 1: hi: 186, btch: 31 usd: 161 [42699.437380] CPU 2: hi: 186, btch: 31 usd: 0 [42699.437383] CPU 3: hi: 186, btch: 31 usd: 34 [42699.437386] CPU 4: hi: 186, btch: 31 usd: 29 [42699.437389] CPU 5: hi: 186, btch: 31 usd: 44 [42699.437392] CPU 6: hi: 186, btch: 31 usd: 68 [42699.437395] CPU 7: hi: 186, btch: 31 usd: 135 [42699.437398] CPU 8: hi: 186, btch: 31 usd: 38 [42699.437401] CPU 9: hi: 186, btch: 31 usd: 24 [42699.437404] CPU 10: hi: 186, btch: 31 usd: 13 [42699.437407] CPU 11: hi: 186, btch: 31 usd: 110 [42699.437410] CPU 12: hi: 186, btch: 31 usd: 186 [42699.437413] CPU 13: hi: 186, btch: 31 usd: 161 [42699.437415] CPU 14: hi: 186, btch: 31 usd: 196 [42699.437418] CPU 15: hi: 186, btch: 31 usd: 179 [42699.437421] CPU 16: hi: 186, btch: 31 usd: 0 [42699.437424] CPU 17: hi: 186, btch: 31 usd: 162 [42699.437427] CPU 18: hi: 186, btch: 31 usd: 18 [42699.437430] CPU 19: hi: 186, btch: 31 usd: 1 [42699.437433] CPU 20: hi: 186, btch: 31 usd: 47 [42699.437436] CPU 21: hi: 186, btch: 31 usd: 164 [42699.437439] CPU 22: hi: 186, btch: 31 usd: 26 [42699.437442] CPU 23: hi: 186, btch: 31 usd: 25 [42699.437445] CPU 24: hi: 186, btch: 31 usd: 122 [42699.437448] CPU 25: hi: 186, btch: 31 usd: 135 [42699.437451] CPU 26: hi: 186, btch: 31 usd: 180 [42699.437454] CPU 27: hi: 186, btch: 31 usd: 110 [42699.437457] CPU 28: hi: 186, btch: 31 usd: 42 [42699.437460] CPU 29: hi: 186, btch: 31 usd: 206 [42699.437462] CPU 30: hi: 186, btch: 31 usd: 175 [42699.437465] CPU 31: hi: 186, btch: 31 usd: 42 [42699.437473] active_anon:174635 inactive_anon:58466 isolated_anon:0 [42699.437475] active_file:33231 inactive_file:8442783 isolated_file:0 [42699.437477] unevictable:2648 dirty:12423 writeback:313 unstable:66930 [42699.437479] free:882251 slab_reclaimable:4383318 slab_unreclaimable:1338402 [42699.437480] mapped:12267 shmem:301 pagetables:3057 bounce:0 [42699.437484] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [42699.437499] lowmem_reserve[]: 0 2959 32249 32249 [42699.437505] Node 0 DMA32 free:122144kB min:4128kB low:5160kB high:6192kB active_anon:104kB inactive_anon:40964kB active_file:300kB inactive_file:558936kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:712404kB slab_unreclaimable:975920kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2 all_unreclaimable? no [42699.437521] lowmem_reserve[]: 0 0 29290 29290 [42699.437526] Node 0 Normal free:54000kB min:40864kB low:51080kB high:61296kB active_anon:520624kB inactive_anon:105900kB active_file:100700kB inactive_file:20293296kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:2944kB writeback:0kB mapped:30864kB shmem:300kB slab_reclaimable:4893464kB slab_unreclaimable:2432588kB kernel_stack:2448kB pagetables:6496kB unstable:186868kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [42699.437544] lowmem_reserve[]: 0 0 0 0 [42699.437548] Node 1 Normal free:3337388kB min:45092kB low:56364kB high:67636kB active_anon:177812kB inactive_anon:87000kB active_file:31924kB inactive_file:12919248kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:46748kB writeback:1404kB mapped:18204kB shmem:904kB slab_reclaimable:11927404kB slab_unreclaimable:1945100kB kernel_stack:3104kB pagetables:5732kB unstable:80852kB bounce:0kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no [42699.437565] lowmem_reserve[]: 0 0 0 0 [42699.437570] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [42699.437583] Node 0 DMA32: 30346*4kB 63*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 121888kB [42699.437596] Node 0 Normal: 12960*4kB 278*8kB 43*16kB 6*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54944kB [42699.437611] Node 1 Normal: 833704*4kB 269*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3336984kB [42699.437623] 8476880 total pagecache pages [42699.437626] 0 pages in swap cache [42699.437629] Swap cache stats: add 0, delete 0, find 0/0 [42699.437631] Free swap = 8388604kB [42699.437633] Total swap = 8388604kB [42699.813363] 16777200 pages RAM [42699.813368] 393967 pages reserved [42699.813370] 3980700 pages shared [42699.813372] 11581648 pages non-shared [42699.813460] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [42699.813484] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315258514 total bytes allocated by lnet [42699.813503] LustreError: 98842:0:(niobuf.c:376:ptlrpc_register_bulk()) export-MDT0000-mdc-ffff88080dd47c00: LNetMDAttach failed x1456758854975224/0: rc = -12 [42699.813559] Lustre: 98842:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880ca8722800 x1456758854975224/t0(0) o37->export-MDT0000-mdc-ffff88080dd47c00@172.18.4.5@o2ib:23/10 lens 448/440 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [42699.813599] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection to export-MDT0000 (at 172.18.4.5@o2ib) was lost; in progress operations using this service will wait for recovery to complete [42699.829251] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection restored to export-MDT0000 (at 172.18.4.5@o2ib) [42700.202054] The following is only an harmless informational message. [42700.202057] Unless you get a _continuous_flood_ of these messages it means [42700.202059] everything is working fine. Allocations from irqs cannot be [42700.202060] perfectly reliable and the kernel is designed to handle that. [42700.202062] find: page allocation failure: order:1, mode:0x40 [42700.202066] Pid: 98842, comm: find Tainted: P NX 3.0.51-0.7.9-default #1 [42700.202068] Call Trace: [42700.202104] [] dump_trace+0x75/0x300 [42700.202118] [] dump_stack+0x69/0x6f [42700.202137] [] warn_alloc_failed+0x102/0x1a0 [42700.202146] [] __alloc_pages_slowpath+0x541/0x7f0 [42700.202153] [] __alloc_pages_nodemask+0x1e9/0x200 [42700.202163] [] kmem_getpages+0x56/0x170 [42700.202171] [] fallback_alloc+0x19b/0x270 [42700.202178] [] __kmalloc+0x285/0x330 [42700.202207] [] LNetMDAttach+0x163/0x5b0 [lnet] [42700.202314] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [42700.202395] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [42700.202473] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [42700.202546] [] ptlrpc_set_wait+0x664/0x8f0 [ptlrpc] [42700.202620] [] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [42700.202660] [] mdc_readpage+0x169/0x630 [mdc] [42700.202683] [] lmv_readpage+0x2dd/0x710 [lmv] [42700.202724] [] ll_dir_filler+0x38c/0xb00 [lustre] [42700.202739] [] __read_cache_page+0x85/0xf0 [42700.202748] [] do_read_cache_page+0x34/0xd0 [42700.202756] [] read_cache_page+0x9/0x20 [42700.202783] [] ll_get_dir_page+0xe83/0x1260 [lustre] [42700.202815] [] ll_dir_read+0x94/0x380 [lustre] [42700.202846] [] ll_readdir+0x16c/0x360 [lustre] [42700.202861] [] vfs_readdir+0xc2/0xe0 [42700.202870] [] sys_getdents64+0x84/0xe0 [42700.202880] [] system_call_fastpath+0x16/0x1b [42700.202923] [<00007f76aed61f9a>] 0x7f76aed61f99 [42700.202926] Mem-Info: [42700.202929] Node 0 DMA per-cpu: [42700.202933] CPU 0: hi: 0, btch: 1 usd: 0 [42700.202936] CPU 1: hi: 0, btch: 1 usd: 0 [42700.202939] CPU 2: hi: 0, btch: 1 usd: 0 [42700.202942] CPU 3: hi: 0, btch: 1 usd: 0 [42700.202945] CPU 4: hi: 0, btch: 1 usd: 0 [42700.202948] CPU 5: hi: 0, btch: 1 usd: 0 [42700.202951] CPU 6: hi: 0, btch: 1 usd: 0 [42700.202954] CPU 7: hi: 0, btch: 1 usd: 0 [42700.202957] CPU 8: hi: 0, btch: 1 usd: 0 [42700.202960] CPU 9: hi: 0, btch: 1 usd: 0 [42700.202963] CPU 10: hi: 0, btch: 1 usd: 0 [42700.202965] CPU 11: hi: 0, btch: 1 usd: 0 [42700.202968] CPU 12: hi: 0, btch: 1 usd: 0 [42700.202971] CPU 13: hi: 0, btch: 1 usd: 0 [42700.202974] CPU 14: hi: 0, btch: 1 usd: 0 [42700.202977] CPU 15: hi: 0, btch: 1 usd: 0 [42700.202980] CPU 16: hi: 0, btch: 1 usd: 0 [42700.202983] CPU 17: hi: 0, btch: 1 usd: 0 [42700.202986] CPU 18: hi: 0, btch: 1 usd: 0 [42700.202988] CPU 19: hi: 0, btch: 1 usd: 0 [42700.202991] CPU 20: hi: 0, btch: 1 usd: 0 [42700.202994] CPU 21: hi: 0, btch: 1 usd: 0 [42700.202997] CPU 22: hi: 0, btch: 1 usd: 0 [42700.203000] CPU 23: hi: 0, btch: 1 usd: 0 [42700.203003] CPU 24: hi: 0, btch: 1 usd: 0 [42700.203006] CPU 25: hi: 0, btch: 1 usd: 0 [42700.203009] CPU 26: hi: 0, btch: 1 usd: 0 [42700.203012] CPU 27: hi: 0, btch: 1 usd: 0 [42700.203015] CPU 28: hi: 0, btch: 1 usd: 0 [42700.203017] CPU 29: hi: 0, btch: 1 usd: 0 [42700.203020] CPU 30: hi: 0, btch: 1 usd: 0 [42700.203023] CPU 31: hi: 0, btch: 1 usd: 0 [42700.203025] Node 0 DMA32 per-cpu: [42700.203029] CPU 0: hi: 186, btch: 31 usd: 159 [42700.203033] CPU 1: hi: 186, btch: 31 usd: 179 [42700.203036] CPU 2: hi: 186, btch: 31 usd: 30 [42700.203039] CPU 3: hi: 186, btch: 31 usd: 0 [42700.203042] CPU 4: hi: 186, btch: 31 usd: 30 [42700.203045] CPU 5: hi: 186, btch: 31 usd: 20 [42700.203048] CPU 6: hi: 186, btch: 31 usd: 26 [42700.203051] CPU 7: hi: 186, btch: 31 usd: 25 [42700.203054] CPU 8: hi: 186, btch: 31 usd: 0 [42700.203057] CPU 9: hi: 186, btch: 31 usd: 0 [42700.203060] CPU 10: hi: 186, btch: 31 usd: 0 [42700.203063] CPU 11: hi: 186, btch: 31 usd: 0 [42700.203066] CPU 12: hi: 186, btch: 31 usd: 135 [42700.203069] CPU 13: hi: 186, btch: 31 usd: 127 [42700.203072] CPU 14: hi: 186, btch: 31 usd: 0 [42700.203074] CPU 15: hi: 186, btch: 31 usd: 105 [42700.203077] CPU 16: hi: 186, btch: 31 usd: 184 [42700.203080] CPU 17: hi: 186, btch: 31 usd: 171 [42700.203084] CPU 18: hi: 186, btch: 31 usd: 91 [42700.203087] CPU 19: hi: 186, btch: 31 usd: 61 [42700.203090] CPU 20: hi: 186, btch: 31 usd: 60 [42700.203093] CPU 21: hi: 186, btch: 31 usd: 155 [42700.203096] CPU 22: hi: 186, btch: 31 usd: 46 [42700.203099] CPU 23: hi: 186, btch: 31 usd: 77 [42700.203101] CPU 24: hi: 186, btch: 31 usd: 0 [42700.203104] CPU 25: hi: 186, btch: 31 usd: 0 [42700.203107] CPU 26: hi: 186, btch: 31 usd: 0 [42700.203110] CPU 27: hi: 186, btch: 31 usd: 0 [42700.203113] CPU 28: hi: 186, btch: 31 usd: 0 [42700.203116] CPU 29: hi: 186, btch: 31 usd: 127 [42700.203119] CPU 30: hi: 186, btch: 31 usd: 53 [42700.203122] CPU 31: hi: 186, btch: 31 usd: 0 [42700.203124] Node 0 Normal per-cpu: [42700.203128] CPU 0: hi: 186, btch: 31 usd: 177 [42700.203131] CPU 1: hi: 186, btch: 31 usd: 155 [42700.203134] CPU 2: hi: 186, btch: 31 usd: 39 [42700.203136] CPU 3: hi: 186, btch: 31 usd: 148 [42700.203139] CPU 4: hi: 186, btch: 31 usd: 27 [42700.203142] CPU 5: hi: 186, btch: 31 usd: 49 [42700.203145] CPU 6: hi: 186, btch: 31 usd: 193 [42700.203148] CPU 7: hi: 186, btch: 31 usd: 170 [42700.203151] CPU 8: hi: 186, btch: 31 usd: 6 [42700.203154] CPU 9: hi: 186, btch: 31 usd: 14 [42700.203156] CPU 10: hi: 186, btch: 31 usd: 14 [42700.203159] CPU 11: hi: 186, btch: 31 usd: 0 [42700.203162] CPU 12: hi: 186, btch: 31 usd: 162 [42700.203165] CPU 13: hi: 186, btch: 31 usd: 177 [42700.203168] CPU 14: hi: 186, btch: 31 usd: 159 [42700.203171] CPU 15: hi: 186, btch: 31 usd: 132 [42700.203174] CPU 16: hi: 186, btch: 31 usd: 17 [42700.203177] CPU 17: hi: 186, btch: 31 usd: 165 [42700.203180] CPU 18: hi: 186, btch: 31 usd: 170 [42700.203182] CPU 19: hi: 186, btch: 31 usd: 172 [42700.203185] CPU 20: hi: 186, btch: 31 usd: 70 [42700.203188] CPU 21: hi: 186, btch: 31 usd: 166 [42700.203191] CPU 22: hi: 186, btch: 31 usd: 169 [42700.203194] CPU 23: hi: 186, btch: 31 usd: 22 [42700.203197] CPU 24: hi: 186, btch: 31 usd: 0 [42700.203200] CPU 25: hi: 186, btch: 31 usd: 185 [42700.203202] CPU 26: hi: 186, btch: 31 usd: 0 [42700.203205] CPU 27: hi: 186, btch: 31 usd: 20 [42700.203208] CPU 28: hi: 186, btch: 31 usd: 0 [42700.203211] CPU 29: hi: 186, btch: 31 usd: 62 [42700.203214] CPU 30: hi: 186, btch: 31 usd: 41 [42700.203217] CPU 31: hi: 186, btch: 31 usd: 0 [42700.203220] Node 1 Normal per-cpu: [42700.203223] CPU 0: hi: 186, btch: 31 usd: 156 [42700.203226] CPU 1: hi: 186, btch: 31 usd: 161 [42700.203229] CPU 2: hi: 186, btch: 31 usd: 0 [42700.203232] CPU 3: hi: 186, btch: 31 usd: 34 [42700.203235] CPU 4: hi: 186, btch: 31 usd: 52 [42700.203238] CPU 5: hi: 186, btch: 31 usd: 33 [42700.203241] CPU 6: hi: 186, btch: 31 usd: 68 [42700.203244] CPU 7: hi: 186, btch: 31 usd: 136 [42700.203247] CPU 8: hi: 186, btch: 31 usd: 46 [42700.203250] CPU 9: hi: 186, btch: 31 usd: 63 [42700.203253] CPU 10: hi: 186, btch: 31 usd: 11 [42700.203256] CPU 11: hi: 186, btch: 31 usd: 108 [42700.203259] CPU 12: hi: 186, btch: 31 usd: 213 [42700.203262] CPU 13: hi: 186, btch: 31 usd: 178 [42700.203265] CPU 14: hi: 186, btch: 31 usd: 183 [42700.203268] CPU 15: hi: 186, btch: 31 usd: 145 [42700.203270] CPU 16: hi: 186, btch: 31 usd: 0 [42700.203273] CPU 17: hi: 186, btch: 31 usd: 162 [42700.203276] CPU 18: hi: 186, btch: 31 usd: 20 [42700.203279] CPU 19: hi: 186, btch: 31 usd: 1 [42700.203282] CPU 20: hi: 186, btch: 31 usd: 47 [42700.203286] CPU 21: hi: 186, btch: 31 usd: 164 [42700.203289] CPU 22: hi: 186, btch: 31 usd: 26 [42700.203292] CPU 23: hi: 186, btch: 31 usd: 25 [42700.203295] CPU 24: hi: 186, btch: 31 usd: 120 [42700.203298] CPU 25: hi: 186, btch: 31 usd: 139 [42700.203300] CPU 26: hi: 186, btch: 31 usd: 185 [42700.203303] CPU 27: hi: 186, btch: 31 usd: 102 [42700.203306] CPU 28: hi: 186, btch: 31 usd: 44 [42700.203309] CPU 29: hi: 186, btch: 31 usd: 204 [42700.203312] CPU 30: hi: 186, btch: 31 usd: 203 [42700.203315] CPU 31: hi: 186, btch: 31 usd: 212 [42700.203323] active_anon:174620 inactive_anon:58466 isolated_anon:0 [42700.203325] active_file:33320 inactive_file:8440391 isolated_file:0 [42700.203326] unevictable:2648 dirty:12021 writeback:333 unstable:81915 [42700.203328] free:897383 slab_reclaimable:4380454 slab_unreclaimable:1339041 [42700.203330] mapped:12267 shmem:301 pagetables:3029 bounce:0 [42700.203333] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [42700.203349] lowmem_reserve[]: 0 2959 32249 32249 [42700.203354] Node 0 DMA32 free:122340kB min:4128kB low:5160kB high:6192kB active_anon:104kB inactive_anon:40964kB active_file:300kB inactive_file:556760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:712404kB slab_unreclaimable:977816kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [42700.203371] lowmem_reserve[]: 0 0 29290 29290 [42700.203388] Node 0 Normal free:54204kB min:40864kB low:51080kB high:61296kB active_anon:520564kB inactive_anon:105900kB active_file:100700kB inactive_file:20294952kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:2732kB writeback:196kB mapped:30864kB shmem:300kB slab_reclaimable:4883560kB slab_unreclaimable:2440404kB kernel_stack:2448kB pagetables:6496kB unstable:228484kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [42700.203398] lowmem_reserve[]: 0 0 0 0 [42700.203401] Node 1 Normal free:3397516kB min:45092kB low:56364kB high:67636kB active_anon:177812kB inactive_anon:87000kB active_file:32280kB inactive_file:12909468kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:45352kB writeback:1136kB mapped:18204kB shmem:904kB slab_reclaimable:11925852kB slab_unreclaimable:1937944kB kernel_stack:3104kB pagetables:5620kB unstable:99176kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [42700.203410] lowmem_reserve[]: 0 0 0 0 [42700.203412] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [42700.203420] Node 0 DMA32: 30555*4kB 15*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 122340kB [42700.203427] Node 0 Normal: 12858*4kB 232*8kB 51*16kB 9*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54392kB [42700.203434] Node 1 Normal: 845736*4kB 1700*8kB 19*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3396848kB [42700.203440] 8474505 total pagecache pages [42700.203442] 0 pages in swap cache [42700.203443] Swap cache stats: add 0, delete 0, find 0/0 [42700.203459] Free swap = 8388604kB [42700.203461] Total swap = 8388604kB [42700.552515] 16777200 pages RAM [42700.552519] 393967 pages reserved [42700.552522] 4008282 pages shared [42700.552523] 11565530 pages non-shared [43128.913429] The following is only an harmless informational message. [43128.913435] Unless you get a _continuous_flood_ of these messages it means [43128.913438] everything is working fine. Allocations from irqs cannot be [43128.913441] perfectly reliable and the kernel is designed to handle that. [43128.913444] find: page allocation failure: order:1, mode:0x40 [43128.913450] Pid: 98842, comm: find Tainted: P NX 3.0.51-0.7.9-default #1 [43128.913454] Call Trace: [43128.913481] [] dump_trace+0x75/0x300 [43128.913495] [] dump_stack+0x69/0x6f [43128.913507] [] warn_alloc_failed+0x102/0x1a0 [43128.913516] [] __alloc_pages_slowpath+0x541/0x7f0 [43128.913524] [] __alloc_pages_nodemask+0x1e9/0x200 [43128.913534] [] kmem_getpages+0x56/0x170 [43128.913542] [] fallback_alloc+0x19b/0x270 [43128.913549] [] __kmalloc+0x285/0x330 [43128.913579] [] LNetMDAttach+0x163/0x5b0 [lnet] [43128.913694] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [43128.913776] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [43128.913854] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [43128.913928] [] ptlrpc_set_wait+0x664/0x8f0 [ptlrpc] [43128.914002] [] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [43128.914044] [] mdc_readpage+0x169/0x630 [mdc] [43128.914066] [] lmv_readpage+0x2dd/0x710 [lmv] [43128.914109] [] ll_dir_filler+0x38c/0xb00 [lustre] [43128.914125] [] __read_cache_page+0x85/0xf0 [43128.914133] [] do_read_cache_page+0x34/0xd0 [43128.914142] [] read_cache_page+0x9/0x20 [43128.914169] [] ll_get_dir_page+0xe83/0x1260 [lustre] [43128.914201] [] ll_dir_read+0x94/0x380 [lustre] [43128.914233] [] ll_readdir+0x16c/0x360 [lustre] [43128.914247] [] vfs_readdir+0xc2/0xe0 [43128.914256] [] sys_getdents64+0x84/0xe0 [43128.914267] [] system_call_fastpath+0x16/0x1b [43128.914314] [<00007f76aed61f9a>] 0x7f76aed61f99 [43128.914317] Mem-Info: [43128.914320] Node 0 DMA per-cpu: [43128.914324] CPU 0: hi: 0, btch: 1 usd: 0 [43128.914328] CPU 1: hi: 0, btch: 1 usd: 0 [43128.914331] CPU 2: hi: 0, btch: 1 usd: 0 [43128.914334] CPU 3: hi: 0, btch: 1 usd: 0 [43128.914337] CPU 4: hi: 0, btch: 1 usd: 0 [43128.914340] CPU 5: hi: 0, btch: 1 usd: 0 [43128.914343] CPU 6: hi: 0, btch: 1 usd: 0 [43128.914346] CPU 7: hi: 0, btch: 1 usd: 0 [43128.914349] CPU 8: hi: 0, btch: 1 usd: 0 [43128.914352] CPU 9: hi: 0, btch: 1 usd: 0 [43128.914355] CPU 10: hi: 0, btch: 1 usd: 0 [43128.914357] CPU 11: hi: 0, btch: 1 usd: 0 [43128.914360] CPU 12: hi: 0, btch: 1 usd: 0 [43128.914363] CPU 13: hi: 0, btch: 1 usd: 0 [43128.914366] CPU 14: hi: 0, btch: 1 usd: 0 [43128.914369] CPU 15: hi: 0, btch: 1 usd: 0 [43128.914372] CPU 16: hi: 0, btch: 1 usd: 0 [43128.914375] CPU 17: hi: 0, btch: 1 usd: 0 [43128.914378] CPU 18: hi: 0, btch: 1 usd: 0 [43128.914381] CPU 19: hi: 0, btch: 1 usd: 0 [43128.914384] CPU 20: hi: 0, btch: 1 usd: 0 [43128.914387] CPU 21: hi: 0, btch: 1 usd: 0 [43128.914389] CPU 22: hi: 0, btch: 1 usd: 0 [43128.914392] CPU 23: hi: 0, btch: 1 usd: 0 [43128.914395] CPU 24: hi: 0, btch: 1 usd: 0 [43128.914398] CPU 25: hi: 0, btch: 1 usd: 0 [43128.914401] CPU 26: hi: 0, btch: 1 usd: 0 [43128.914404] CPU 27: hi: 0, btch: 1 usd: 0 [43128.914407] CPU 28: hi: 0, btch: 1 usd: 0 [43128.914410] CPU 29: hi: 0, btch: 1 usd: 0 [43128.914412] CPU 30: hi: 0, btch: 1 usd: 0 [43128.914415] CPU 31: hi: 0, btch: 1 usd: 0 [43128.914418] Node 0 DMA32 per-cpu: [43128.914422] CPU 0: hi: 186, btch: 31 usd: 170 [43128.914425] CPU 1: hi: 186, btch: 31 usd: 173 [43128.914428] CPU 2: hi: 186, btch: 31 usd: 35 [43128.914431] CPU 3: hi: 186, btch: 31 usd: 25 [43128.914434] CPU 4: hi: 186, btch: 31 usd: 26 [43128.914437] CPU 5: hi: 186, btch: 31 usd: 23 [43128.914440] CPU 6: hi: 186, btch: 31 usd: 30 [43128.914443] CPU 7: hi: 186, btch: 31 usd: 2 [43128.914446] CPU 8: hi: 186, btch: 31 usd: 1 [43128.914449] CPU 9: hi: 186, btch: 31 usd: 0 [43128.914452] CPU 10: hi: 186, btch: 31 usd: 0 [43128.914455] CPU 11: hi: 186, btch: 31 usd: 0 [43128.914458] CPU 12: hi: 186, btch: 31 usd: 158 [43128.914461] CPU 13: hi: 186, btch: 31 usd: 0 [43128.914464] CPU 14: hi: 186, btch: 31 usd: 0 [43128.914467] CPU 15: hi: 186, btch: 31 usd: 4 [43128.914470] CPU 16: hi: 186, btch: 31 usd: 161 [43128.914473] CPU 17: hi: 186, btch: 31 usd: 148 [43128.914476] CPU 18: hi: 186, btch: 31 usd: 179 [43128.914479] CPU 19: hi: 186, btch: 31 usd: 38 [43128.914482] CPU 20: hi: 186, btch: 31 usd: 10 [43128.914485] CPU 21: hi: 186, btch: 31 usd: 78 [43128.914488] CPU 22: hi: 186, btch: 31 usd: 66 [43128.914491] CPU 23: hi: 186, btch: 31 usd: 51 [43128.914494] CPU 24: hi: 186, btch: 31 usd: 0 [43128.914497] CPU 25: hi: 186, btch: 31 usd: 0 [43128.914500] CPU 26: hi: 186, btch: 31 usd: 0 [43128.914502] CPU 27: hi: 186, btch: 31 usd: 17 [43128.914505] CPU 28: hi: 186, btch: 31 usd: 177 [43128.914508] CPU 29: hi: 186, btch: 31 usd: 61 [43128.914512] CPU 30: hi: 186, btch: 31 usd: 0 [43128.914514] CPU 31: hi: 186, btch: 31 usd: 20 [43128.914517] Node 0 Normal per-cpu: [43128.914520] CPU 0: hi: 186, btch: 31 usd: 183 [43128.914523] CPU 1: hi: 186, btch: 31 usd: 164 [43128.914526] CPU 2: hi: 186, btch: 31 usd: 172 [43128.914529] CPU 3: hi: 186, btch: 31 usd: 162 [43128.914531] CPU 4: hi: 186, btch: 31 usd: 171 [43128.914534] CPU 5: hi: 186, btch: 31 usd: 44 [43128.914537] CPU 6: hi: 186, btch: 31 usd: 179 [43128.914540] CPU 7: hi: 186, btch: 31 usd: 111 [43128.914543] CPU 8: hi: 186, btch: 31 usd: 116 [43128.914546] CPU 9: hi: 186, btch: 31 usd: 65 [43128.914549] CPU 10: hi: 186, btch: 31 usd: 130 [43128.914551] CPU 11: hi: 186, btch: 31 usd: 155 [43128.914554] CPU 12: hi: 186, btch: 31 usd: 160 [43128.914557] CPU 13: hi: 186, btch: 31 usd: 139 [43128.914560] CPU 14: hi: 186, btch: 31 usd: 13 [43128.914563] CPU 15: hi: 186, btch: 31 usd: 11 [43128.914566] CPU 16: hi: 186, btch: 31 usd: 210 [43128.914569] CPU 17: hi: 186, btch: 31 usd: 205 [43128.914572] CPU 18: hi: 186, btch: 31 usd: 168 [43128.914575] CPU 19: hi: 186, btch: 31 usd: 157 [43128.914577] CPU 20: hi: 186, btch: 31 usd: 203 [43128.914580] CPU 21: hi: 186, btch: 31 usd: 154 [43128.914583] CPU 22: hi: 186, btch: 31 usd: 173 [43128.914586] CPU 23: hi: 186, btch: 31 usd: 175 [43128.914589] CPU 24: hi: 186, btch: 31 usd: 0 [43128.914592] CPU 25: hi: 186, btch: 31 usd: 0 [43128.914595] CPU 26: hi: 186, btch: 31 usd: 0 [43128.914597] CPU 27: hi: 186, btch: 31 usd: 157 [43128.914600] CPU 28: hi: 186, btch: 31 usd: 0 [43128.914603] CPU 29: hi: 186, btch: 31 usd: 0 [43128.914606] CPU 30: hi: 186, btch: 31 usd: 3 [43128.914609] CPU 31: hi: 186, btch: 31 usd: 1 [43128.914612] Node 1 Normal per-cpu: [43128.914615] CPU 0: hi: 186, btch: 31 usd: 172 [43128.914618] CPU 1: hi: 186, btch: 31 usd: 166 [43128.914621] CPU 2: hi: 186, btch: 31 usd: 178 [43128.914624] CPU 3: hi: 186, btch: 31 usd: 162 [43128.914628] CPU 4: hi: 186, btch: 31 usd: 171 [43128.914631] CPU 5: hi: 186, btch: 31 usd: 142 [43128.914634] CPU 6: hi: 186, btch: 31 usd: 172 [43128.914637] CPU 7: hi: 186, btch: 31 usd: 46 [43128.914640] CPU 8: hi: 186, btch: 31 usd: 24 [43128.914643] CPU 9: hi: 186, btch: 31 usd: 28 [43128.914646] CPU 10: hi: 186, btch: 31 usd: 161 [43128.914649] CPU 11: hi: 186, btch: 31 usd: 168 [43128.914652] CPU 12: hi: 186, btch: 31 usd: 186 [43128.914655] CPU 13: hi: 186, btch: 31 usd: 161 [43128.914658] CPU 14: hi: 186, btch: 31 usd: 204 [43128.914661] CPU 15: hi: 186, btch: 31 usd: 180 [43128.914664] CPU 16: hi: 186, btch: 31 usd: 179 [43128.914667] CPU 17: hi: 186, btch: 31 usd: 62 [43128.914670] CPU 18: hi: 186, btch: 31 usd: 7 [43128.914673] CPU 19: hi: 186, btch: 31 usd: 0 [43128.914676] CPU 20: hi: 186, btch: 31 usd: 169 [43128.914679] CPU 21: hi: 186, btch: 31 usd: 38 [43128.914682] CPU 22: hi: 186, btch: 31 usd: 0 [43128.914685] CPU 23: hi: 186, btch: 31 usd: 0 [43128.914688] CPU 24: hi: 186, btch: 31 usd: 17 [43128.914691] CPU 25: hi: 186, btch: 31 usd: 181 [43128.914694] CPU 26: hi: 186, btch: 31 usd: 127 [43128.914697] CPU 27: hi: 186, btch: 31 usd: 162 [43128.914700] CPU 28: hi: 186, btch: 31 usd: 156 [43128.914703] CPU 29: hi: 186, btch: 31 usd: 140 [43128.914706] CPU 30: hi: 186, btch: 31 usd: 34 [43128.914709] CPU 31: hi: 186, btch: 31 usd: 180 [43128.914730] active_anon:188938 inactive_anon:58465 isolated_anon:0 [43128.914731] active_file:44598 inactive_file:9717261 isolated_file:33 [43128.914732] unevictable:2648 dirty:10346 writeback:119 unstable:56204 [43128.914733] free:623111 slab_reclaimable:3612979 slab_unreclaimable:1369091 [43128.914734] mapped:12270 shmem:301 pagetables:3057 bounce:0 [43128.914736] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [43128.914745] lowmem_reserve[]: 0 2959 32249 32249 [43128.914748] Node 0 DMA32 free:229648kB min:4128kB low:5160kB high:6192kB active_anon:2148kB inactive_anon:40964kB active_file:452kB inactive_file:1220080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:375144kB slab_unreclaimable:542992kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [43128.914757] lowmem_reserve[]: 0 0 29290 29290 [43128.914760] Node 0 Normal free:56160kB min:40864kB low:51080kB high:61296kB active_anon:582056kB inactive_anon:105900kB active_file:151660kB inactive_file:20278780kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:480kB writeback:452kB mapped:30872kB shmem:300kB slab_reclaimable:4972708kB slab_unreclaimable:2246832kB kernel_stack:2448kB pagetables:6496kB unstable:107560kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [43128.914769] lowmem_reserve[]: 0 0 0 0 [43128.914772] Node 1 Normal free:2191164kB min:45092kB low:56364kB high:67636kB active_anon:171548kB inactive_anon:86996kB active_file:26280kB inactive_file:17370184kB unevictable:2064kB isolated(anon):0kB isolated(file):132kB present:33095680kB mlocked:2064kB dirty:40904kB writeback:24kB mapped:18208kB shmem:904kB slab_reclaimable:9104064kB slab_unreclaimable:2686540kB kernel_stack:3112kB pagetables:5732kB unstable:117256kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [43128.914797] lowmem_reserve[]: 0 0 0 0 [43128.914803] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [43128.914816] Node 0 DMA32: 57048*4kB 178*8kB 5*16kB 3*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 229792kB [43128.914829] Node 0 Normal: 13132*4kB 423*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 55912kB [43128.914842] Node 1 Normal: 543930*4kB 1918*8kB 19*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2191368kB [43128.914859] 9763111 total pagecache pages [43128.914861] 0 pages in swap cache [43128.914864] Swap cache stats: add 0, delete 0, find 0/0 [43128.914867] Free swap = 8388604kB [43128.914869] Total swap = 8388604kB [43129.265661] 16777200 pages RAM [43129.265665] 393967 pages reserved [43129.265668] 3147316 pages shared [43129.265669] 12664329 pages non-shared [43129.265713] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [43129.265732] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) Skipped 1 previous similar message [43129.265746] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315259106 total bytes allocated by lnet [43129.265760] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) Skipped 1 previous similar message [43129.265778] LustreError: 98842:0:(niobuf.c:376:ptlrpc_register_bulk()) export-MDT0000-mdc-ffff88080dd47c00: LNetMDAttach failed x1456758877817408/0: rc = -12 [43129.265793] LustreError: 98842:0:(niobuf.c:376:ptlrpc_register_bulk()) Skipped 1 previous similar message [43129.265846] Lustre: 98842:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880dbdc81c00 x1456758877817408/t0(0) o37->export-MDT0000-mdc-ffff88080dd47c00@172.18.4.5@o2ib:23/10 lens 448/440 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [43129.265872] Lustre: 98842:0:(client.c:1903:ptlrpc_expire_one_request()) Skipped 1 previous similar message [43129.265894] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection to export-MDT0000 (at 172.18.4.5@o2ib) was lost; in progress operations using this service will wait for recovery to complete [43129.265911] Lustre: Skipped 1 previous similar message [43129.281415] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection restored to export-MDT0000 (at 172.18.4.5@o2ib) [43129.281422] Lustre: Skipped 1 previous similar message [43140.825150] The following is only an harmless informational message. [43140.825156] Unless you get a _continuous_flood_ of these messages it means [43140.825159] everything is working fine. Allocations from irqs cannot be [43140.825161] perfectly reliable and the kernel is designed to handle that. [43140.825165] find: page allocation failure: order:1, mode:0x40 [43140.825171] Pid: 98842, comm: find Tainted: P NX 3.0.51-0.7.9-default #1 [43140.825175] Call Trace: [43140.825202] [] dump_trace+0x75/0x300 [43140.825235] [] dump_stack+0x69/0x6f [43140.825242] [] warn_alloc_failed+0x102/0x1a0 [43140.825248] [] __alloc_pages_slowpath+0x541/0x7f0 [43140.825252] [] __alloc_pages_nodemask+0x1e9/0x200 [43140.825258] [] kmem_getpages+0x56/0x170 [43140.825262] [] fallback_alloc+0x19b/0x270 [43140.825266] [] __kmalloc+0x285/0x330 [43140.825286] [] LNetMDAttach+0x163/0x5b0 [lnet] [43140.825361] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [43140.825406] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [43140.825449] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [43140.825490] [] ptlrpc_set_wait+0x664/0x8f0 [ptlrpc] [43140.825530] [] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [43140.825553] [] mdc_readpage+0x169/0x630 [mdc] [43140.825566] [] lmv_readpage+0x2dd/0x710 [lmv] [43140.825591] [] ll_dir_filler+0x38c/0xb00 [lustre] [43140.825600] [] __read_cache_page+0x85/0xf0 [43140.825605] [] do_read_cache_page+0x34/0xd0 [43140.825609] [] read_cache_page+0x9/0x20 [43140.825624] [] ll_get_dir_page+0xe83/0x1260 [lustre] [43140.825642] [] ll_dir_read+0x94/0x380 [lustre] [43140.825659] [] ll_readdir+0x16c/0x360 [lustre] [43140.825668] [] vfs_readdir+0xc2/0xe0 [43140.825672] [] sys_getdents64+0x84/0xe0 [43140.825679] [] system_call_fastpath+0x16/0x1b [43140.825710] [<00007f76aed61f9a>] 0x7f76aed61f99 [43140.825712] Mem-Info: [43140.825713] Node 0 DMA per-cpu: [43140.825716] CPU 0: hi: 0, btch: 1 usd: 0 [43140.825718] CPU 1: hi: 0, btch: 1 usd: 0 [43140.825719] CPU 2: hi: 0, btch: 1 usd: 0 [43140.825721] CPU 3: hi: 0, btch: 1 usd: 0 [43140.825723] CPU 4: hi: 0, btch: 1 usd: 0 [43140.825724] CPU 5: hi: 0, btch: 1 usd: 0 [43140.825726] CPU 6: hi: 0, btch: 1 usd: 0 [43140.825728] CPU 7: hi: 0, btch: 1 usd: 0 [43140.825729] CPU 8: hi: 0, btch: 1 usd: 0 [43140.825731] CPU 9: hi: 0, btch: 1 usd: 0 [43140.825732] CPU 10: hi: 0, btch: 1 usd: 0 [43140.825734] CPU 11: hi: 0, btch: 1 usd: 0 [43140.825736] CPU 12: hi: 0, btch: 1 usd: 0 [43140.825737] CPU 13: hi: 0, btch: 1 usd: 0 [43140.825739] CPU 14: hi: 0, btch: 1 usd: 0 [43140.825740] CPU 15: hi: 0, btch: 1 usd: 0 [43140.825742] CPU 16: hi: 0, btch: 1 usd: 0 [43140.825744] CPU 17: hi: 0, btch: 1 usd: 0 [43140.825745] CPU 18: hi: 0, btch: 1 usd: 0 [43140.825747] CPU 19: hi: 0, btch: 1 usd: 0 [43140.825748] CPU 20: hi: 0, btch: 1 usd: 0 [43140.825750] CPU 21: hi: 0, btch: 1 usd: 0 [43140.825752] CPU 22: hi: 0, btch: 1 usd: 0 [43140.825753] CPU 23: hi: 0, btch: 1 usd: 0 [43140.825755] CPU 24: hi: 0, btch: 1 usd: 0 [43140.825757] CPU 25: hi: 0, btch: 1 usd: 0 [43140.825758] CPU 26: hi: 0, btch: 1 usd: 0 [43140.825760] CPU 27: hi: 0, btch: 1 usd: 0 [43140.825761] CPU 28: hi: 0, btch: 1 usd: 0 [43140.825763] CPU 29: hi: 0, btch: 1 usd: 0 [43140.825765] CPU 30: hi: 0, btch: 1 usd: 0 [43140.825766] CPU 31: hi: 0, btch: 1 usd: 0 [43140.825767] Node 0 DMA32 per-cpu: [43140.825770] CPU 0: hi: 186, btch: 31 usd: 175 [43140.825772] CPU 1: hi: 186, btch: 31 usd: 208 [43140.825773] CPU 2: hi: 186, btch: 31 usd: 13 [43140.825775] CPU 3: hi: 186, btch: 31 usd: 16 [43140.825777] CPU 4: hi: 186, btch: 31 usd: 175 [43140.825778] CPU 5: hi: 186, btch: 31 usd: 15 [43140.825780] CPU 6: hi: 186, btch: 31 usd: 24 [43140.825782] CPU 7: hi: 186, btch: 31 usd: 2 [43140.825783] CPU 8: hi: 186, btch: 31 usd: 0 [43140.825785] CPU 9: hi: 186, btch: 31 usd: 0 [43140.825787] CPU 10: hi: 186, btch: 31 usd: 0 [43140.825788] CPU 11: hi: 186, btch: 31 usd: 0 [43140.825790] CPU 12: hi: 186, btch: 31 usd: 105 [43140.825791] CPU 13: hi: 186, btch: 31 usd: 0 [43140.825793] CPU 14: hi: 186, btch: 31 usd: 0 [43140.825795] CPU 15: hi: 186, btch: 31 usd: 4 [43140.825796] CPU 16: hi: 186, btch: 31 usd: 157 [43140.825798] CPU 17: hi: 186, btch: 31 usd: 166 [43140.825800] CPU 18: hi: 186, btch: 31 usd: 141 [43140.825801] CPU 19: hi: 186, btch: 31 usd: 63 [43140.825803] CPU 20: hi: 186, btch: 31 usd: 38 [43140.825805] CPU 21: hi: 186, btch: 31 usd: 85 [43140.825806] CPU 22: hi: 186, btch: 31 usd: 66 [43140.825808] CPU 23: hi: 186, btch: 31 usd: 51 [43140.825810] CPU 24: hi: 186, btch: 31 usd: 0 [43140.825811] CPU 25: hi: 186, btch: 31 usd: 0 [43140.825813] CPU 26: hi: 186, btch: 31 usd: 0 [43140.825815] CPU 27: hi: 186, btch: 31 usd: 0 [43140.825816] CPU 28: hi: 186, btch: 31 usd: 146 [43140.825818] CPU 29: hi: 186, btch: 31 usd: 61 [43140.825819] CPU 30: hi: 186, btch: 31 usd: 1 [43140.825821] CPU 31: hi: 186, btch: 31 usd: 1 [43140.825822] Node 0 Normal per-cpu: [43140.825824] CPU 0: hi: 186, btch: 31 usd: 169 [43140.825826] CPU 1: hi: 186, btch: 31 usd: 164 [43140.825827] CPU 2: hi: 186, btch: 31 usd: 159 [43140.825829] CPU 3: hi: 186, btch: 31 usd: 59 [43140.825831] CPU 4: hi: 186, btch: 31 usd: 177 [43140.825832] CPU 5: hi: 186, btch: 31 usd: 43 [43140.825834] CPU 6: hi: 186, btch: 31 usd: 175 [43140.825835] CPU 7: hi: 186, btch: 31 usd: 111 [43140.825837] CPU 8: hi: 186, btch: 31 usd: 180 [43140.825838] CPU 9: hi: 186, btch: 31 usd: 65 [43140.825840] CPU 10: hi: 186, btch: 31 usd: 108 [43140.825842] CPU 11: hi: 186, btch: 31 usd: 155 [43140.825843] CPU 12: hi: 186, btch: 31 usd: 173 [43140.825845] CPU 13: hi: 186, btch: 31 usd: 108 [43140.825846] CPU 14: hi: 186, btch: 31 usd: 1 [43140.825848] CPU 15: hi: 186, btch: 31 usd: 0 [43140.825849] CPU 16: hi: 186, btch: 31 usd: 172 [43140.825851] CPU 17: hi: 186, btch: 31 usd: 180 [43140.825852] CPU 18: hi: 186, btch: 31 usd: 167 [43140.825854] CPU 19: hi: 186, btch: 31 usd: 178 [43140.825856] CPU 20: hi: 186, btch: 31 usd: 177 [43140.825857] CPU 21: hi: 186, btch: 31 usd: 163 [43140.825859] CPU 22: hi: 186, btch: 31 usd: 174 [43140.825860] CPU 23: hi: 186, btch: 31 usd: 175 [43140.825862] CPU 24: hi: 186, btch: 31 usd: 11 [43140.825863] CPU 25: hi: 186, btch: 31 usd: 21 [43140.825865] CPU 26: hi: 186, btch: 31 usd: 0 [43140.825867] CPU 27: hi: 186, btch: 31 usd: 176 [43140.825868] CPU 28: hi: 186, btch: 31 usd: 0 [43140.825870] CPU 29: hi: 186, btch: 31 usd: 0 [43140.825871] CPU 30: hi: 186, btch: 31 usd: 1 [43140.825873] CPU 31: hi: 186, btch: 31 usd: 13 [43140.825874] Node 1 Normal per-cpu: [43140.825876] CPU 0: hi: 186, btch: 31 usd: 161 [43140.825878] CPU 1: hi: 186, btch: 31 usd: 176 [43140.825879] CPU 2: hi: 186, btch: 31 usd: 50 [43140.825881] CPU 3: hi: 186, btch: 31 usd: 31 [43140.825883] CPU 4: hi: 186, btch: 31 usd: 158 [43140.825884] CPU 5: hi: 186, btch: 31 usd: 33 [43140.825886] CPU 6: hi: 186, btch: 31 usd: 181 [43140.825888] CPU 7: hi: 186, btch: 31 usd: 36 [43140.825889] CPU 8: hi: 186, btch: 31 usd: 182 [43140.825891] CPU 9: hi: 186, btch: 31 usd: 64 [43140.825893] CPU 10: hi: 186, btch: 31 usd: 120 [43140.825894] CPU 11: hi: 186, btch: 31 usd: 175 [43140.825896] CPU 12: hi: 186, btch: 31 usd: 176 [43140.825897] CPU 13: hi: 186, btch: 31 usd: 29 [43140.825899] CPU 14: hi: 186, btch: 31 usd: 38 [43140.825901] CPU 15: hi: 186, btch: 31 usd: 166 [43140.825902] CPU 16: hi: 186, btch: 31 usd: 165 [43140.825904] CPU 17: hi: 186, btch: 31 usd: 176 [43140.825906] CPU 18: hi: 186, btch: 31 usd: 48 [43140.825907] CPU 19: hi: 186, btch: 31 usd: 29 [43140.825909] CPU 20: hi: 186, btch: 31 usd: 20 [43140.825911] CPU 21: hi: 186, btch: 31 usd: 62 [43140.825912] CPU 22: hi: 186, btch: 31 usd: 30 [43140.825914] CPU 23: hi: 186, btch: 31 usd: 0 [43140.825915] CPU 24: hi: 186, btch: 31 usd: 29 [43140.825917] CPU 25: hi: 186, btch: 31 usd: 148 [43140.825919] CPU 26: hi: 186, btch: 31 usd: 127 [43140.825920] CPU 27: hi: 186, btch: 31 usd: 173 [43140.825922] CPU 28: hi: 186, btch: 31 usd: 52 [43140.825923] CPU 29: hi: 186, btch: 31 usd: 181 [43140.825925] CPU 30: hi: 186, btch: 31 usd: 197 [43140.825927] CPU 31: hi: 186, btch: 31 usd: 57 [43140.825931] active_anon:188936 inactive_anon:58465 isolated_anon:0 [43140.825932] active_file:44796 inactive_file:10011595 isolated_file:0 [43140.825933] unevictable:2648 dirty:17555 writeback:214 unstable:91164 [43140.825934] free:591380 slab_reclaimable:3342952 slab_unreclaimable:1406765 [43140.825935] mapped:12270 shmem:301 pagetables:3057 bounce:0 [43140.825937] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [43140.825947] lowmem_reserve[]: 0 2959 32249 32249 [43140.825950] Node 0 DMA32 free:122712kB min:4128kB low:5160kB high:6192kB active_anon:2148kB inactive_anon:40964kB active_file:452kB inactive_file:1189900kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:373056kB slab_unreclaimable:682176kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:8 all_unreclaimable? no [43140.825959] lowmem_reserve[]: 0 0 29290 29290 [43140.825961] Node 0 Normal free:51700kB min:40864kB low:51080kB high:61296kB active_anon:582004kB inactive_anon:105900kB active_file:151888kB inactive_file:20334812kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:1212kB writeback:0kB mapped:30872kB shmem:300kB slab_reclaimable:4908172kB slab_unreclaimable:2259368kB kernel_stack:2448kB pagetables:6496kB unstable:118880kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [43140.825971] lowmem_reserve[]: 0 0 0 0 [43140.825974] Node 1 Normal free:2175636kB min:45092kB low:56364kB high:67636kB active_anon:171592kB inactive_anon:86996kB active_file:26844kB inactive_file:18520892kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:69008kB writeback:1316kB mapped:18208kB shmem:904kB slab_reclaimable:8090580kB slab_unreclaimable:2685516kB kernel_stack:3112kB pagetables:5732kB unstable:245776kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [43140.825983] lowmem_reserve[]: 0 0 0 0 [43140.825986] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [43140.825993] Node 0 DMA32: 30571*4kB 12*8kB 6*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 122476kB [43140.826000] Node 0 Normal: 12807*4kB 119*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52196kB [43140.826007] Node 1 Normal: 540218*4kB 1511*8kB 93*16kB 10*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2174768kB [43140.826014] 10057023 total pagecache pages [43140.826015] 0 pages in swap cache [43140.826017] Swap cache stats: add 0, delete 0, find 0/0 [43140.826018] Free swap = 8388604kB [43140.826020] Total swap = 8388604kB [43141.148785] 16777200 pages RAM [43141.148789] 393967 pages reserved [43141.148791] 3400481 pages shared [43141.148793] 12481660 pages non-shared [43141.148852] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [43141.148872] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315259106 total bytes allocated by lnet [43141.148890] LustreError: 98842:0:(niobuf.c:376:ptlrpc_register_bulk()) export-MDT0000-mdc-ffff88080dd47c00: LNetMDAttach failed x1456758877941172/0: rc = -12 [43141.148939] Lustre: 98842:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff88098a014c00 x1456758877941172/t0(0) o37->export-MDT0000-mdc-ffff88080dd47c00@172.18.4.5@o2ib:23/10 lens 448/440 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [43141.148968] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection to export-MDT0000 (at 172.18.4.5@o2ib) was lost; in progress operations using this service will wait for recovery to complete [43141.149584] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection restored to export-MDT0000 (at 172.18.4.5@o2ib) [43218.690059] The following is only an harmless informational message. [43218.690063] Unless you get a _continuous_flood_ of these messages it means [43218.690064] everything is working fine. Allocations from irqs cannot be [43218.690066] perfectly reliable and the kernel is designed to handle that. [43218.690068] find: page allocation failure: order:1, mode:0x40 [43218.690071] Pid: 98842, comm: find Tainted: P NX 3.0.51-0.7.9-default #1 [43218.690074] Call Trace: [43218.690093] [] dump_trace+0x75/0x300 [43218.690102] [] dump_stack+0x69/0x6f [43218.690109] [] warn_alloc_failed+0x102/0x1a0 [43218.690115] [] __alloc_pages_slowpath+0x541/0x7f0 [43218.690119] [] __alloc_pages_nodemask+0x1e9/0x200 [43218.690125] [] kmem_getpages+0x56/0x170 [43218.690130] [] fallback_alloc+0x19b/0x270 [43218.690134] [] __kmalloc+0x285/0x330 [43218.690155] [] LNetMDAttach+0x163/0x5b0 [lnet] [43218.690238] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [43218.690283] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [43218.690326] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [43218.690367] [] ptlrpc_set_wait+0x664/0x8f0 [ptlrpc] [43218.690407] [] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [43218.690430] [] mdc_readpage+0x169/0x630 [mdc] [43218.690446] [] lmv_readpage+0x2dd/0x710 [lmv] [43218.690475] [] ll_dir_filler+0x38c/0xb00 [lustre] [43218.690485] [] __read_cache_page+0x85/0xf0 [43218.690490] [] do_read_cache_page+0x34/0xd0 [43218.690494] [] read_cache_page+0x9/0x20 [43218.690509] [] ll_get_dir_page+0xe83/0x1260 [lustre] [43218.690526] [] ll_dir_read+0x94/0x380 [lustre] [43218.690561] [] ll_readdir+0x16c/0x360 [lustre] [43218.690576] [] vfs_readdir+0xc2/0xe0 [43218.690585] [] sys_getdents64+0x84/0xe0 [43218.690596] [] system_call_fastpath+0x16/0x1b [43218.690646] [<00007f76aed61f9a>] 0x7f76aed61f99 [43218.690649] Mem-Info: [43218.690652] Node 0 DMA per-cpu: [43218.690656] CPU 0: hi: 0, btch: 1 usd: 0 [43218.690660] CPU 1: hi: 0, btch: 1 usd: 0 [43218.690663] CPU 2: hi: 0, btch: 1 usd: 0 [43218.690666] CPU 3: hi: 0, btch: 1 usd: 0 [43218.690669] CPU 4: hi: 0, btch: 1 usd: 0 [43218.690672] CPU 5: hi: 0, btch: 1 usd: 0 [43218.690675] CPU 6: hi: 0, btch: 1 usd: 0 [43218.690678] CPU 7: hi: 0, btch: 1 usd: 0 [43218.690680] CPU 8: hi: 0, btch: 1 usd: 0 [43218.690683] CPU 9: hi: 0, btch: 1 usd: 0 [43218.690686] CPU 10: hi: 0, btch: 1 usd: 0 [43218.690689] CPU 11: hi: 0, btch: 1 usd: 0 [43218.690692] CPU 12: hi: 0, btch: 1 usd: 0 [43218.690695] CPU 13: hi: 0, btch: 1 usd: 0 [43218.690698] CPU 14: hi: 0, btch: 1 usd: 0 [43218.690701] CPU 15: hi: 0, btch: 1 usd: 0 [43218.690703] CPU 16: hi: 0, btch: 1 usd: 0 [43218.690706] CPU 17: hi: 0, btch: 1 usd: 0 [43218.690709] CPU 18: hi: 0, btch: 1 usd: 0 [43218.690712] CPU 19: hi: 0, btch: 1 usd: 0 [43218.690715] CPU 20: hi: 0, btch: 1 usd: 0 [43218.690718] CPU 21: hi: 0, btch: 1 usd: 0 [43218.690721] CPU 22: hi: 0, btch: 1 usd: 0 [43218.690724] CPU 23: hi: 0, btch: 1 usd: 0 [43218.690727] CPU 24: hi: 0, btch: 1 usd: 0 [43218.690730] CPU 25: hi: 0, btch: 1 usd: 0 [43218.690733] CPU 26: hi: 0, btch: 1 usd: 0 [43218.690736] CPU 27: hi: 0, btch: 1 usd: 0 [43218.690738] CPU 28: hi: 0, btch: 1 usd: 0 [43218.690741] CPU 29: hi: 0, btch: 1 usd: 0 [43218.690744] CPU 30: hi: 0, btch: 1 usd: 0 [43218.690747] CPU 31: hi: 0, btch: 1 usd: 0 [43218.690750] Node 0 DMA32 per-cpu: [43218.690754] CPU 0: hi: 186, btch: 31 usd: 189 [43218.690757] CPU 1: hi: 186, btch: 31 usd: 151 [43218.690760] CPU 2: hi: 186, btch: 31 usd: 194 [43218.690763] CPU 3: hi: 186, btch: 31 usd: 0 [43218.690766] CPU 4: hi: 186, btch: 31 usd: 160 [43218.690770] CPU 5: hi: 186, btch: 31 usd: 3 [43218.690773] CPU 6: hi: 186, btch: 31 usd: 166 [43218.690776] CPU 7: hi: 186, btch: 31 usd: 165 [43218.690779] CPU 8: hi: 186, btch: 31 usd: 23 [43218.690782] CPU 9: hi: 186, btch: 31 usd: 65 [43218.690785] CPU 10: hi: 186, btch: 31 usd: 0 [43218.690788] CPU 11: hi: 186, btch: 31 usd: 30 [43218.690791] CPU 12: hi: 186, btch: 31 usd: 30 [43218.690794] CPU 13: hi: 186, btch: 31 usd: 54 [43218.690797] CPU 14: hi: 186, btch: 31 usd: 198 [43218.690800] CPU 15: hi: 186, btch: 31 usd: 56 [43218.690803] CPU 16: hi: 186, btch: 31 usd: 171 [43218.690806] CPU 17: hi: 186, btch: 31 usd: 193 [43218.690809] CPU 18: hi: 186, btch: 31 usd: 43 [43218.690812] CPU 19: hi: 186, btch: 31 usd: 167 [43218.690816] CPU 20: hi: 186, btch: 31 usd: 184 [43218.690819] CPU 21: hi: 186, btch: 31 usd: 174 [43218.690822] CPU 22: hi: 186, btch: 31 usd: 184 [43218.690825] CPU 23: hi: 186, btch: 31 usd: 176 [43218.690828] CPU 24: hi: 186, btch: 31 usd: 129 [43218.690831] CPU 25: hi: 186, btch: 31 usd: 0 [43218.690834] CPU 26: hi: 186, btch: 31 usd: 168 [43218.690837] CPU 27: hi: 186, btch: 31 usd: 0 [43218.690840] CPU 28: hi: 186, btch: 31 usd: 179 [43218.690843] CPU 29: hi: 186, btch: 31 usd: 0 [43218.690846] CPU 30: hi: 186, btch: 31 usd: 179 [43218.690849] CPU 31: hi: 186, btch: 31 usd: 51 [43218.690851] Node 0 Normal per-cpu: [43218.690855] CPU 0: hi: 186, btch: 31 usd: 180 [43218.690857] CPU 1: hi: 186, btch: 31 usd: 47 [43218.690860] CPU 2: hi: 186, btch: 31 usd: 43 [43218.690864] CPU 3: hi: 186, btch: 31 usd: 30 [43218.690866] CPU 4: hi: 186, btch: 31 usd: 63 [43218.690869] CPU 5: hi: 186, btch: 31 usd: 26 [43218.690872] CPU 6: hi: 186, btch: 31 usd: 50 [43218.690875] CPU 7: hi: 186, btch: 31 usd: 163 [43218.690878] CPU 8: hi: 186, btch: 31 usd: 83 [43218.690881] CPU 9: hi: 186, btch: 31 usd: 93 [43218.690883] CPU 10: hi: 186, btch: 31 usd: 75 [43218.690886] CPU 11: hi: 186, btch: 31 usd: 29 [43218.690889] CPU 12: hi: 186, btch: 31 usd: 180 [43218.690892] CPU 13: hi: 186, btch: 31 usd: 211 [43218.690895] CPU 14: hi: 186, btch: 31 usd: 216 [43218.690898] CPU 15: hi: 186, btch: 31 usd: 204 [43218.690901] CPU 16: hi: 186, btch: 31 usd: 49 [43218.690904] CPU 17: hi: 186, btch: 31 usd: 123 [43218.690907] CPU 18: hi: 186, btch: 31 usd: 122 [43218.690910] CPU 19: hi: 186, btch: 31 usd: 57 [43218.690913] CPU 20: hi: 186, btch: 31 usd: 185 [43218.690915] CPU 21: hi: 186, btch: 31 usd: 174 [43218.690918] CPU 22: hi: 186, btch: 31 usd: 168 [43218.690921] CPU 23: hi: 186, btch: 31 usd: 162 [43218.690924] CPU 24: hi: 186, btch: 31 usd: 55 [43218.690927] CPU 25: hi: 186, btch: 31 usd: 33 [43218.690930] CPU 26: hi: 186, btch: 31 usd: 173 [43218.690932] CPU 27: hi: 186, btch: 31 usd: 0 [43218.690935] CPU 28: hi: 186, btch: 31 usd: 154 [43218.690938] CPU 29: hi: 186, btch: 31 usd: 0 [43218.690941] CPU 30: hi: 186, btch: 31 usd: 188 [43218.690944] CPU 31: hi: 186, btch: 31 usd: 201 [43218.690947] Node 1 Normal per-cpu: [43218.690950] CPU 0: hi: 186, btch: 31 usd: 174 [43218.690953] CPU 1: hi: 186, btch: 31 usd: 146 [43218.690956] CPU 2: hi: 186, btch: 31 usd: 14 [43218.690959] CPU 3: hi: 186, btch: 31 usd: 0 [43218.690962] CPU 4: hi: 186, btch: 31 usd: 163 [43218.690965] CPU 5: hi: 186, btch: 31 usd: 167 [43218.690968] CPU 6: hi: 186, btch: 31 usd: 120 [43218.690971] CPU 7: hi: 186, btch: 31 usd: 185 [43218.690974] CPU 8: hi: 186, btch: 31 usd: 46 [43218.690977] CPU 9: hi: 186, btch: 31 usd: 27 [43218.690980] CPU 10: hi: 186, btch: 31 usd: 158 [43218.690983] CPU 11: hi: 186, btch: 31 usd: 158 [43218.690986] CPU 12: hi: 186, btch: 31 usd: 168 [43218.690989] CPU 13: hi: 186, btch: 31 usd: 162 [43218.690991] CPU 14: hi: 186, btch: 31 usd: 161 [43218.690994] CPU 15: hi: 186, btch: 31 usd: 204 [43218.690997] CPU 16: hi: 186, btch: 31 usd: 9 [43218.691000] CPU 17: hi: 186, btch: 31 usd: 162 [43218.691003] CPU 18: hi: 186, btch: 31 usd: 48 [43218.691006] CPU 19: hi: 186, btch: 31 usd: 164 [43218.691009] CPU 20: hi: 186, btch: 31 usd: 166 [43218.691012] CPU 21: hi: 186, btch: 31 usd: 167 [43218.691016] CPU 22: hi: 186, btch: 31 usd: 91 [43218.691019] CPU 23: hi: 186, btch: 31 usd: 105 [43218.691022] CPU 24: hi: 186, btch: 31 usd: 29 [43218.691025] CPU 25: hi: 186, btch: 31 usd: 19 [43218.691027] CPU 26: hi: 186, btch: 31 usd: 160 [43218.691030] CPU 27: hi: 186, btch: 31 usd: 168 [43218.691033] CPU 28: hi: 186, btch: 31 usd: 40 [43218.691036] CPU 29: hi: 186, btch: 31 usd: 184 [43218.691039] CPU 30: hi: 186, btch: 31 usd: 166 [43218.691042] CPU 31: hi: 186, btch: 31 usd: 184 [43218.691050] active_anon:188955 inactive_anon:58465 isolated_anon:0 [43218.691052] active_file:48117 inactive_file:11227776 isolated_file:19 [43218.691054] unevictable:2648 dirty:1615 writeback:2243 unstable:78815 [43218.691056] free:117628 slab_reclaimable:2653969 slab_unreclaimable:1478563 [43218.691057] mapped:12270 shmem:301 pagetables:3057 bounce:0 [43218.691061] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [43218.691076] lowmem_reserve[]: 0 2959 32249 32249 [43218.691082] Node 0 DMA32 free:163656kB min:4128kB low:5160kB high:6192kB active_anon:2148kB inactive_anon:40964kB active_file:496kB inactive_file:1073040kB unevictable:0kB isolated(anon):0kB isolated(file):76kB present:3030348kB mlocked:0kB dirty:428kB writeback:444kB mapped:0kB shmem:0kB slab_reclaimable:301544kB slab_unreclaimable:820460kB kernel_stack:0kB pagetables:0kB unstable:16160kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [43218.691099] lowmem_reserve[]: 0 0 29290 29290 [43218.691104] Node 0 Normal free:232888kB min:40864kB low:51080kB high:61296kB active_anon:582108kB inactive_anon:105904kB active_file:154232kB inactive_file:21392548kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:1936kB writeback:8848kB mapped:30872kB shmem:304kB slab_reclaimable:3807984kB slab_unreclaimable:2259708kB kernel_stack:2440kB pagetables:6496kB unstable:72560kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [43218.691121] lowmem_reserve[]: 0 0 0 0 [43218.691126] Node 1 Normal free:58868kB min:45092kB low:56364kB high:67636kB active_anon:171564kB inactive_anon:86992kB active_file:37740kB inactive_file:22445124kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:4096kB writeback:0kB mapped:18208kB shmem:900kB slab_reclaimable:6506348kB slab_unreclaimable:2834084kB kernel_stack:3112kB pagetables:5732kB unstable:226540kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [43218.691143] lowmem_reserve[]: 0 0 0 0 [43218.691148] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [43218.691161] Node 0 DMA32: 40870*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 163504kB [43218.691175] Node 0 Normal: 57996*4kB 82*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 232640kB [43218.691187] Node 1 Normal: 14426*4kB 307*8kB 4*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60224kB [43218.691200] 11276482 total pagecache pages [43218.691202] 0 pages in swap cache [43218.691205] Swap cache stats: add 0, delete 0, find 0/0 [43218.691208] Free swap = 8388604kB [43218.691210] Total swap = 8388604kB [43218.975675] 16777200 pages RAM [43218.975678] 393967 pages reserved [43218.975679] 4147733 pages shared [43218.975680] 12189814 pages non-shared [43218.975739] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [43218.975752] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315259330 total bytes allocated by lnet [43218.975763] LustreError: 98842:0:(niobuf.c:376:ptlrpc_register_bulk()) export-MDT0000-mdc-ffff88080dd47c00: LNetMDAttach failed x1456758879336612/0: rc = -12 [43218.975798] Lustre: 98842:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff8803c8990c00 x1456758879336612/t0(0) o37->export-MDT0000-mdc-ffff88080dd47c00@172.18.4.5@o2ib:23/10 lens 448/440 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [43218.975818] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection to export-MDT0000 (at 172.18.4.5@o2ib) was lost; in progress operations using this service will wait for recovery to complete [43218.976301] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection restored to export-MDT0000 (at 172.18.4.5@o2ib) [43339.890895] The following is only an harmless informational message. [43339.890901] Unless you get a _continuous_flood_ of these messages it means [43339.890904] everything is working fine. Allocations from irqs cannot be [43339.890906] perfectly reliable and the kernel is designed to handle that. [43339.890910] find: page allocation failure: order:1, mode:0x40 [43339.890916] Pid: 98842, comm: find Tainted: P NX 3.0.51-0.7.9-default #1 [43339.890920] Call Trace: [43339.890949] [] dump_trace+0x75/0x300 [43339.890963] [] dump_stack+0x69/0x6f [43339.890975] [] warn_alloc_failed+0x102/0x1a0 [43339.890984] [] __alloc_pages_slowpath+0x541/0x7f0 [43339.890992] [] __alloc_pages_nodemask+0x1e9/0x200 [43339.891002] [] kmem_getpages+0x56/0x170 [43339.891011] [] fallback_alloc+0x19b/0x270 [43339.891019] [] __kmalloc+0x285/0x330 [43339.891049] [] LNetMDAttach+0x163/0x5b0 [lnet] [43339.891167] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [43339.891249] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [43339.891328] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [43339.891406] [] ptlrpc_set_wait+0x664/0x8f0 [ptlrpc] [43339.891483] [] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [43339.891525] [] mdc_readpage+0x169/0x630 [mdc] [43339.891549] [] lmv_readpage+0x2dd/0x710 [lmv] [43339.891593] [] ll_dir_filler+0x38c/0xb00 [lustre] [43339.891609] [] __read_cache_page+0x85/0xf0 [43339.891617] [] do_read_cache_page+0x34/0xd0 [43339.891626] [] read_cache_page+0x9/0x20 [43339.891653] [] ll_get_dir_page+0xe83/0x1260 [lustre] [43339.891685] [] ll_dir_read+0x94/0x380 [lustre] [43339.891716] [] ll_readdir+0x16c/0x360 [lustre] [43339.891731] [] vfs_readdir+0xc2/0xe0 [43339.891740] [] sys_getdents64+0x84/0xe0 [43339.891750] [] system_call_fastpath+0x16/0x1b [43339.891800] [<00007f76aed61f9a>] 0x7f76aed61f99 [43339.891803] Mem-Info: [43339.891805] Node 0 DMA per-cpu: [43339.891810] CPU 0: hi: 0, btch: 1 usd: 0 [43339.891813] CPU 1: hi: 0, btch: 1 usd: 0 [43339.891816] CPU 2: hi: 0, btch: 1 usd: 0 [43339.891819] CPU 3: hi: 0, btch: 1 usd: 0 [43339.891822] CPU 4: hi: 0, btch: 1 usd: 0 [43339.891825] CPU 5: hi: 0, btch: 1 usd: 0 [43339.891828] CPU 6: hi: 0, btch: 1 usd: 0 [43339.891831] CPU 7: hi: 0, btch: 1 usd: 0 [43339.891834] CPU 8: hi: 0, btch: 1 usd: 0 [43339.891837] CPU 9: hi: 0, btch: 1 usd: 0 [43339.891840] CPU 10: hi: 0, btch: 1 usd: 0 [43339.891843] CPU 11: hi: 0, btch: 1 usd: 0 [43339.891846] CPU 12: hi: 0, btch: 1 usd: 0 [43339.891849] CPU 13: hi: 0, btch: 1 usd: 0 [43339.891852] CPU 14: hi: 0, btch: 1 usd: 0 [43339.891855] CPU 15: hi: 0, btch: 1 usd: 0 [43339.891858] CPU 16: hi: 0, btch: 1 usd: 0 [43339.891861] CPU 17: hi: 0, btch: 1 usd: 0 [43339.891864] CPU 18: hi: 0, btch: 1 usd: 0 [43339.891867] CPU 19: hi: 0, btch: 1 usd: 0 [43339.891869] CPU 20: hi: 0, btch: 1 usd: 0 [43339.891872] CPU 21: hi: 0, btch: 1 usd: 0 [43339.891875] CPU 22: hi: 0, btch: 1 usd: 0 [43339.891878] CPU 23: hi: 0, btch: 1 usd: 0 [43339.891881] CPU 24: hi: 0, btch: 1 usd: 0 [43339.891884] CPU 25: hi: 0, btch: 1 usd: 0 [43339.891887] CPU 26: hi: 0, btch: 1 usd: 0 [43339.891890] CPU 27: hi: 0, btch: 1 usd: 0 [43339.891893] CPU 28: hi: 0, btch: 1 usd: 0 [43339.891896] CPU 29: hi: 0, btch: 1 usd: 0 [43339.891899] CPU 30: hi: 0, btch: 1 usd: 0 [43339.891901] CPU 31: hi: 0, btch: 1 usd: 0 [43339.891904] Node 0 DMA32 per-cpu: [43339.891908] CPU 0: hi: 186, btch: 31 usd: 180 [43339.891911] CPU 1: hi: 186, btch: 31 usd: 168 [43339.891914] CPU 2: hi: 186, btch: 31 usd: 68 [43339.891918] CPU 3: hi: 186, btch: 31 usd: 0 [43339.891921] CPU 4: hi: 186, btch: 31 usd: 22 [43339.891924] CPU 5: hi: 186, btch: 31 usd: 30 [43339.891927] CPU 6: hi: 186, btch: 31 usd: 68 [43339.891930] CPU 7: hi: 186, btch: 31 usd: 0 [43339.891933] CPU 8: hi: 186, btch: 31 usd: 7 [43339.891936] CPU 9: hi: 186, btch: 31 usd: 16 [43339.891939] CPU 10: hi: 186, btch: 31 usd: 0 [43339.891942] CPU 11: hi: 186, btch: 31 usd: 0 [43339.891945] CPU 12: hi: 186, btch: 31 usd: 133 [43339.891948] CPU 13: hi: 186, btch: 31 usd: 53 [43339.891951] CPU 14: hi: 186, btch: 31 usd: 169 [43339.891954] CPU 15: hi: 186, btch: 31 usd: 167 [43339.891957] CPU 16: hi: 186, btch: 31 usd: 186 [43339.891960] CPU 17: hi: 186, btch: 31 usd: 185 [43339.891963] CPU 18: hi: 186, btch: 31 usd: 185 [43339.891966] CPU 19: hi: 186, btch: 31 usd: 158 [43339.891969] CPU 20: hi: 186, btch: 31 usd: 148 [43339.891972] CPU 21: hi: 186, btch: 31 usd: 158 [43339.891975] CPU 22: hi: 186, btch: 31 usd: 0 [43339.891978] CPU 23: hi: 186, btch: 31 usd: 183 [43339.891981] CPU 24: hi: 186, btch: 31 usd: 3 [43339.891984] CPU 25: hi: 186, btch: 31 usd: 0 [43339.891987] CPU 26: hi: 186, btch: 31 usd: 107 [43339.891990] CPU 27: hi: 186, btch: 31 usd: 0 [43339.891993] CPU 28: hi: 186, btch: 31 usd: 96 [43339.891995] CPU 29: hi: 186, btch: 31 usd: 0 [43339.891998] CPU 30: hi: 186, btch: 31 usd: 0 [43339.892001] CPU 31: hi: 186, btch: 31 usd: 0 [43339.892004] Node 0 Normal per-cpu: [43339.892007] CPU 0: hi: 186, btch: 31 usd: 160 [43339.892010] CPU 1: hi: 186, btch: 31 usd: 175 [43339.892013] CPU 2: hi: 186, btch: 31 usd: 156 [43339.892016] CPU 3: hi: 186, btch: 31 usd: 32 [43339.892019] CPU 4: hi: 186, btch: 31 usd: 109 [43339.892022] CPU 5: hi: 186, btch: 31 usd: 57 [43339.892025] CPU 6: hi: 186, btch: 31 usd: 171 [43339.892027] CPU 7: hi: 186, btch: 31 usd: 30 [43339.892030] CPU 8: hi: 186, btch: 31 usd: 2 [43339.892033] CPU 9: hi: 186, btch: 31 usd: 141 [43339.892036] CPU 10: hi: 186, btch: 31 usd: 93 [43339.892039] CPU 11: hi: 186, btch: 31 usd: 135 [43339.892042] CPU 12: hi: 186, btch: 31 usd: 184 [43339.892045] CPU 13: hi: 186, btch: 31 usd: 161 [43339.892048] CPU 14: hi: 186, btch: 31 usd: 155 [43339.892051] CPU 15: hi: 186, btch: 31 usd: 185 [43339.892054] CPU 16: hi: 186, btch: 31 usd: 156 [43339.892057] CPU 17: hi: 186, btch: 31 usd: 169 [43339.892060] CPU 18: hi: 186, btch: 31 usd: 180 [43339.892063] CPU 19: hi: 186, btch: 31 usd: 172 [43339.892066] CPU 20: hi: 186, btch: 31 usd: 150 [43339.892069] CPU 21: hi: 186, btch: 31 usd: 146 [43339.892071] CPU 22: hi: 186, btch: 31 usd: 43 [43339.892074] CPU 23: hi: 186, btch: 31 usd: 180 [43339.892077] CPU 24: hi: 186, btch: 31 usd: 64 [43339.892080] CPU 25: hi: 186, btch: 31 usd: 72 [43339.892083] CPU 26: hi: 186, btch: 31 usd: 109 [43339.892086] CPU 27: hi: 186, btch: 31 usd: 0 [43339.892089] CPU 28: hi: 186, btch: 31 usd: 119 [43339.892092] CPU 29: hi: 186, btch: 31 usd: 5 [43339.892094] CPU 30: hi: 186, btch: 31 usd: 0 [43339.892097] CPU 31: hi: 186, btch: 31 usd: 182 [43339.892100] Node 1 Normal per-cpu: [43339.892103] CPU 0: hi: 186, btch: 31 usd: 171 [43339.892106] CPU 1: hi: 186, btch: 31 usd: 171 [43339.892109] CPU 2: hi: 186, btch: 31 usd: 161 [43339.892112] CPU 3: hi: 186, btch: 31 usd: 1 [43339.892115] CPU 4: hi: 186, btch: 31 usd: 20 [43339.892118] CPU 5: hi: 186, btch: 31 usd: 24 [43339.892122] CPU 6: hi: 186, btch: 31 usd: 156 [43339.892125] CPU 7: hi: 186, btch: 31 usd: 1 [43339.892128] CPU 8: hi: 186, btch: 31 usd: 170 [43339.892131] CPU 9: hi: 186, btch: 31 usd: 139 [43339.892134] CPU 10: hi: 186, btch: 31 usd: 152 [43339.892137] CPU 11: hi: 186, btch: 31 usd: 149 [43339.892140] CPU 12: hi: 186, btch: 31 usd: 40 [43339.892143] CPU 13: hi: 186, btch: 31 usd: 193 [43339.892146] CPU 14: hi: 186, btch: 31 usd: 178 [43339.892149] CPU 15: hi: 186, btch: 31 usd: 185 [43339.892152] CPU 16: hi: 186, btch: 31 usd: 15 [43339.892155] CPU 17: hi: 186, btch: 31 usd: 18 [43339.892158] CPU 18: hi: 186, btch: 31 usd: 172 [43339.892161] CPU 19: hi: 186, btch: 31 usd: 175 [43339.892165] CPU 20: hi: 186, btch: 31 usd: 154 [43339.892168] CPU 21: hi: 186, btch: 31 usd: 170 [43339.892171] CPU 22: hi: 186, btch: 31 usd: 0 [43339.892174] CPU 23: hi: 186, btch: 31 usd: 164 [43339.892177] CPU 24: hi: 186, btch: 31 usd: 154 [43339.892180] CPU 25: hi: 186, btch: 31 usd: 135 [43339.892183] CPU 26: hi: 186, btch: 31 usd: 173 [43339.892186] CPU 27: hi: 186, btch: 31 usd: 174 [43339.892189] CPU 28: hi: 186, btch: 31 usd: 179 [43339.892192] CPU 29: hi: 186, btch: 31 usd: 134 [43339.892195] CPU 30: hi: 186, btch: 31 usd: 5 [43339.892198] CPU 31: hi: 186, btch: 31 usd: 173 [43339.892206] active_anon:188949 inactive_anon:58465 isolated_anon:0 [43339.892208] active_file:52441 inactive_file:12537981 isolated_file:1 [43339.892210] unevictable:2648 dirty:1572 writeback:318 unstable:79798 [43339.892211] free:74958 slab_reclaimable:1355458 slab_unreclaimable:1661272 [43339.892213] mapped:12270 shmem:301 pagetables:3029 bounce:0 [43339.892217] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [43339.892232] lowmem_reserve[]: 0 2959 32249 32249 [43339.892238] Node 0 DMA32 free:122128kB min:4128kB low:5160kB high:6192kB active_anon:2148kB inactive_anon:40964kB active_file:604kB inactive_file:782800kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:290636kB slab_unreclaimable:1168912kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:14 all_unreclaimable? no [43339.892254] lowmem_reserve[]: 0 0 29290 29290 [43339.892259] Node 0 Normal free:55868kB min:40864kB low:51080kB high:61296kB active_anon:582104kB inactive_anon:105904kB active_file:156996kB inactive_file:22084876kB unevictable:8528kB isolated(anon):0kB isolated(file):4kB present:29992960kB mlocked:2396kB dirty:436kB writeback:320kB mapped:30872kB shmem:304kB slab_reclaimable:3288020kB slab_unreclaimable:2581476kB kernel_stack:2440kB pagetables:6384kB unstable:135900kB bounce:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? no [43339.892277] lowmem_reserve[]: 0 0 0 0 [43339.892282] Node 1 Normal free:105992kB min:45092kB low:56364kB high:67636kB active_anon:171544kB inactive_anon:86992kB active_file:52164kB inactive_file:27283860kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:5852kB writeback:952kB mapped:18208kB shmem:900kB slab_reclaimable:1843176kB slab_unreclaimable:2894700kB kernel_stack:3112kB pagetables:5732kB unstable:183292kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no [43339.892299] lowmem_reserve[]: 0 0 0 0 [43339.892303] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [43339.892316] Node 0 DMA32: 30539*4kB 10*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 122252kB [43339.892329] Node 0 Normal: 12998*4kB 147*8kB 104*16kB 28*32kB 14*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56624kB [43339.892343] Node 1 Normal: 21854*4kB 2133*8kB 48*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 105248kB [43339.892357] 12591228 total pagecache pages [43339.892359] 0 pages in swap cache [43339.892362] Swap cache stats: add 0, delete 0, find 0/0 [43339.892365] Free swap = 8388604kB [43339.892367] Total swap = 8388604kB [43340.272343] 16777200 pages RAM [43340.272347] 393967 pages reserved [43340.272349] 5403648 pages shared [43340.272351] 10987899 pages non-shared [43340.272443] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [43340.272464] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315259106 total bytes allocated by lnet [43340.272483] LustreError: 98842:0:(niobuf.c:376:ptlrpc_register_bulk()) export-MDT0000-mdc-ffff88080dd47c00: LNetMDAttach failed x1456758881096012/0: rc = -12 [43340.272536] Lustre: 98842:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880f4ea64000 x1456758881096012/t0(0) o37->export-MDT0000-mdc-ffff88080dd47c00@172.18.4.5@o2ib:23/10 lens 448/440 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [43340.272569] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection to export-MDT0000 (at 172.18.4.5@o2ib) was lost; in progress operations using this service will wait for recovery to complete [43340.288115] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection restored to export-MDT0000 (at 172.18.4.5@o2ib) [43370.198362] The following is only an harmless informational message. [43370.198368] Unless you get a _continuous_flood_ of these messages it means [43370.198371] everything is working fine. Allocations from irqs cannot be [43370.198373] perfectly reliable and the kernel is designed to handle that. [43370.198377] find: page allocation failure: order:1, mode:0x40 [43370.198382] Pid: 98842, comm: find Tainted: P NX 3.0.51-0.7.9-default #1 [43370.198387] Call Trace: [43370.198415] [] dump_trace+0x75/0x300 [43370.198429] [] dump_stack+0x69/0x6f [43370.198440] [] warn_alloc_failed+0x102/0x1a0 [43370.198450] [] __alloc_pages_slowpath+0x541/0x7f0 [43370.198458] [] __alloc_pages_nodemask+0x1e9/0x200 [43370.198468] [] kmem_getpages+0x56/0x170 [43370.198476] [] fallback_alloc+0x19b/0x270 [43370.198484] [] __kmalloc+0x285/0x330 [43370.198514] [] LNetMDAttach+0x163/0x5b0 [lnet] [43370.198627] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [43370.198709] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [43370.198787] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [43370.198861] [] ptlrpc_set_wait+0x664/0x8f0 [ptlrpc] [43370.198935] [] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [43370.198977] [] mdc_readpage+0x169/0x630 [mdc] [43370.198999] [] lmv_readpage+0x2dd/0x710 [lmv] [43370.199040] [] ll_dir_filler+0x38c/0xb00 [lustre] [43370.199055] [] __read_cache_page+0x85/0xf0 [43370.199064] [] do_read_cache_page+0x34/0xd0 [43370.199072] [] read_cache_page+0x9/0x20 [43370.199100] [] ll_get_dir_page+0xe83/0x1260 [lustre] [43370.199132] [] ll_dir_read+0x94/0x380 [lustre] [43370.199163] [] ll_readdir+0x16c/0x360 [lustre] [43370.199178] [] vfs_readdir+0xc2/0xe0 [43370.199187] [] sys_getdents64+0x84/0xe0 [43370.199198] [] system_call_fastpath+0x16/0x1b [43370.199244] [<00007f76aed61f9a>] 0x7f76aed61f99 [43370.199246] Mem-Info: [43370.199249] Node 0 DMA per-cpu: [43370.199253] CPU 0: hi: 0, btch: 1 usd: 0 [43370.199257] CPU 1: hi: 0, btch: 1 usd: 0 [43370.199260] CPU 2: hi: 0, btch: 1 usd: 0 [43370.199263] CPU 3: hi: 0, btch: 1 usd: 0 [43370.199266] CPU 4: hi: 0, btch: 1 usd: 0 [43370.199269] CPU 5: hi: 0, btch: 1 usd: 0 [43370.199272] CPU 6: hi: 0, btch: 1 usd: 0 [43370.199274] CPU 7: hi: 0, btch: 1 usd: 0 [43370.199277] CPU 8: hi: 0, btch: 1 usd: 0 [43370.199280] CPU 9: hi: 0, btch: 1 usd: 0 [43370.199283] CPU 10: hi: 0, btch: 1 usd: 0 [43370.199286] CPU 11: hi: 0, btch: 1 usd: 0 [43370.199289] CPU 12: hi: 0, btch: 1 usd: 0 [43370.199292] CPU 13: hi: 0, btch: 1 usd: 0 [43370.199295] CPU 14: hi: 0, btch: 1 usd: 0 [43370.199298] CPU 15: hi: 0, btch: 1 usd: 0 [43370.199301] CPU 16: hi: 0, btch: 1 usd: 0 [43370.199304] CPU 17: hi: 0, btch: 1 usd: 0 [43370.199307] CPU 18: hi: 0, btch: 1 usd: 0 [43370.199309] CPU 19: hi: 0, btch: 1 usd: 0 [43370.199312] CPU 20: hi: 0, btch: 1 usd: 0 [43370.199315] CPU 21: hi: 0, btch: 1 usd: 0 [43370.199318] CPU 22: hi: 0, btch: 1 usd: 0 [43370.199321] CPU 23: hi: 0, btch: 1 usd: 0 [43370.199324] CPU 24: hi: 0, btch: 1 usd: 0 [43370.199327] CPU 25: hi: 0, btch: 1 usd: 0 [43370.199330] CPU 26: hi: 0, btch: 1 usd: 0 [43370.199333] CPU 27: hi: 0, btch: 1 usd: 0 [43370.199336] CPU 28: hi: 0, btch: 1 usd: 0 [43370.199339] CPU 29: hi: 0, btch: 1 usd: 0 [43370.199341] CPU 30: hi: 0, btch: 1 usd: 0 [43370.199344] CPU 31: hi: 0, btch: 1 usd: 0 [43370.199347] Node 0 DMA32 per-cpu: [43370.199351] CPU 0: hi: 186, btch: 31 usd: 33 [43370.199354] CPU 1: hi: 186, btch: 31 usd: 75 [43370.199357] CPU 2: hi: 186, btch: 31 usd: 1 [43370.199360] CPU 3: hi: 186, btch: 31 usd: 0 [43370.199363] CPU 4: hi: 186, btch: 31 usd: 74 [43370.199366] CPU 5: hi: 186, btch: 31 usd: 165 [43370.199369] CPU 6: hi: 186, btch: 31 usd: 185 [43370.199372] CPU 7: hi: 186, btch: 31 usd: 176 [43370.199375] CPU 8: hi: 186, btch: 31 usd: 0 [43370.199378] CPU 9: hi: 186, btch: 31 usd: 0 [43370.199381] CPU 10: hi: 186, btch: 31 usd: 0 [43370.199384] CPU 11: hi: 186, btch: 31 usd: 0 [43370.199387] CPU 12: hi: 186, btch: 31 usd: 154 [43370.199390] CPU 13: hi: 186, btch: 31 usd: 4 [43370.199393] CPU 14: hi: 186, btch: 31 usd: 165 [43370.199396] CPU 15: hi: 186, btch: 31 usd: 138 [43370.199399] CPU 16: hi: 186, btch: 31 usd: 155 [43370.199402] CPU 17: hi: 186, btch: 31 usd: 185 [43370.199405] CPU 18: hi: 186, btch: 31 usd: 184 [43370.199408] CPU 19: hi: 186, btch: 31 usd: 160 [43370.199411] CPU 20: hi: 186, btch: 31 usd: 175 [43370.199414] CPU 21: hi: 186, btch: 31 usd: 0 [43370.199417] CPU 22: hi: 186, btch: 31 usd: 19 [43370.199420] CPU 23: hi: 186, btch: 31 usd: 180 [43370.199423] CPU 24: hi: 186, btch: 31 usd: 0 [43370.199426] CPU 25: hi: 186, btch: 31 usd: 0 [43370.199429] CPU 26: hi: 186, btch: 31 usd: 107 [43370.199432] CPU 27: hi: 186, btch: 31 usd: 0 [43370.199435] CPU 28: hi: 186, btch: 31 usd: 0 [43370.199437] CPU 29: hi: 186, btch: 31 usd: 0 [43370.199440] CPU 30: hi: 186, btch: 31 usd: 0 [43370.199443] CPU 31: hi: 186, btch: 31 usd: 127 [43370.199445] Node 0 Normal per-cpu: [43370.199449] CPU 0: hi: 186, btch: 31 usd: 55 [43370.199452] CPU 1: hi: 186, btch: 31 usd: 177 [43370.199455] CPU 2: hi: 186, btch: 31 usd: 27 [43370.199458] CPU 3: hi: 186, btch: 31 usd: 26 [43370.199461] CPU 4: hi: 186, btch: 31 usd: 179 [43370.199464] CPU 5: hi: 186, btch: 31 usd: 177 [43370.199467] CPU 6: hi: 186, btch: 31 usd: 169 [43370.199470] CPU 7: hi: 186, btch: 31 usd: 163 [43370.199473] CPU 8: hi: 186, btch: 31 usd: 134 [43370.199476] CPU 9: hi: 186, btch: 31 usd: 52 [43370.199479] CPU 10: hi: 186, btch: 31 usd: 118 [43370.199483] CPU 11: hi: 186, btch: 31 usd: 153 [43370.199485] CPU 12: hi: 186, btch: 31 usd: 170 [43370.199488] CPU 13: hi: 186, btch: 31 usd: 174 [43370.199492] CPU 14: hi: 186, btch: 31 usd: 160 [43370.199494] CPU 15: hi: 186, btch: 31 usd: 174 [43370.199498] CPU 16: hi: 186, btch: 31 usd: 168 [43370.199501] CPU 17: hi: 186, btch: 31 usd: 162 [43370.199504] CPU 18: hi: 186, btch: 31 usd: 157 [43370.199507] CPU 19: hi: 186, btch: 31 usd: 176 [43370.199510] CPU 20: hi: 186, btch: 31 usd: 126 [43370.199513] CPU 21: hi: 186, btch: 31 usd: 8 [43370.199516] CPU 22: hi: 186, btch: 31 usd: 36 [43370.199519] CPU 23: hi: 186, btch: 31 usd: 172 [43370.199522] CPU 24: hi: 186, btch: 31 usd: 9 [43370.199525] CPU 25: hi: 186, btch: 31 usd: 135 [43370.199528] CPU 26: hi: 186, btch: 31 usd: 78 [43370.199531] CPU 27: hi: 186, btch: 31 usd: 0 [43370.199534] CPU 28: hi: 186, btch: 31 usd: 132 [43370.199537] CPU 29: hi: 186, btch: 31 usd: 17 [43370.199540] CPU 30: hi: 186, btch: 31 usd: 25 [43370.199543] CPU 31: hi: 186, btch: 31 usd: 144 [43370.199546] Node 1 Normal per-cpu: [43370.199549] CPU 0: hi: 186, btch: 31 usd: 1 [43370.199553] CPU 1: hi: 186, btch: 31 usd: 172 [43370.199556] CPU 2: hi: 186, btch: 31 usd: 28 [43370.199559] CPU 3: hi: 186, btch: 31 usd: 1 [43370.199562] CPU 4: hi: 186, btch: 31 usd: 9 [43370.199565] CPU 5: hi: 186, btch: 31 usd: 184 [43370.199568] CPU 6: hi: 186, btch: 31 usd: 185 [43370.199571] CPU 7: hi: 186, btch: 31 usd: 163 [43370.199574] CPU 8: hi: 186, btch: 31 usd: 31 [43370.199576] CPU 9: hi: 186, btch: 31 usd: 121 [43370.199579] CPU 10: hi: 186, btch: 31 usd: 157 [43370.199582] CPU 11: hi: 186, btch: 31 usd: 160 [43370.199585] CPU 12: hi: 186, btch: 31 usd: 190 [43370.199588] CPU 13: hi: 186, btch: 31 usd: 52 [43370.199591] CPU 14: hi: 186, btch: 31 usd: 163 [43370.199594] CPU 15: hi: 186, btch: 31 usd: 158 [43370.199597] CPU 16: hi: 186, btch: 31 usd: 4 [43370.199600] CPU 17: hi: 186, btch: 31 usd: 165 [43370.199603] CPU 18: hi: 186, btch: 31 usd: 163 [43370.199606] CPU 19: hi: 186, btch: 31 usd: 175 [43370.199609] CPU 20: hi: 186, btch: 31 usd: 101 [43370.199612] CPU 21: hi: 186, btch: 31 usd: 31 [43370.199615] CPU 22: hi: 186, btch: 31 usd: 52 [43370.199618] CPU 23: hi: 186, btch: 31 usd: 182 [43370.199621] CPU 24: hi: 186, btch: 31 usd: 153 [43370.199624] CPU 25: hi: 186, btch: 31 usd: 74 [43370.199627] CPU 26: hi: 186, btch: 31 usd: 172 [43370.199630] CPU 27: hi: 186, btch: 31 usd: 164 [43370.199633] CPU 28: hi: 186, btch: 31 usd: 189 [43370.199636] CPU 29: hi: 186, btch: 31 usd: 187 [43370.199639] CPU 30: hi: 186, btch: 31 usd: 41 [43370.199641] CPU 31: hi: 186, btch: 31 usd: 174 [43370.199650] active_anon:188960 inactive_anon:58465 isolated_anon:0 [43370.199652] active_file:54859 inactive_file:12677543 isolated_file:32 [43370.199653] unevictable:2648 dirty:3221 writeback:605 unstable:85134 [43370.199655] free:114826 slab_reclaimable:1080150 slab_unreclaimable:1778118 [43370.199657] mapped:12270 shmem:301 pagetables:3100 bounce:0 [43370.199660] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [43370.199677] lowmem_reserve[]: 0 2959 32249 32249 [43370.199682] Node 0 DMA32 free:123476kB min:4128kB low:5160kB high:6192kB active_anon:2148kB inactive_anon:40964kB active_file:604kB inactive_file:693408kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:274456kB slab_unreclaimable:1273072kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:41 all_unreclaimable? no [43370.199699] lowmem_reserve[]: 0 0 29290 29290 [43370.199704] Node 0 Normal free:196720kB min:40864kB low:51080kB high:61296kB active_anon:581784kB inactive_anon:105904kB active_file:163980kB inactive_file:22466360kB unevictable:8528kB isolated(anon):0kB isolated(file):128kB present:29992960kB mlocked:2396kB dirty:496kB writeback:2376kB mapped:30872kB shmem:304kB slab_reclaimable:2528624kB slab_unreclaimable:2812876kB kernel_stack:2432kB pagetables:6408kB unstable:201836kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [43370.199722] lowmem_reserve[]: 0 0 0 0 [43370.199726] Node 1 Normal free:123636kB min:45092kB low:56364kB high:67636kB active_anon:171908kB inactive_anon:86992kB active_file:54852kB inactive_file:27550020kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:12388kB writeback:44kB mapped:18208kB shmem:900kB slab_reclaimable:1517520kB slab_unreclaimable:3026524kB kernel_stack:3120kB pagetables:5992kB unstable:138700kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [43370.199743] lowmem_reserve[]: 0 0 0 0 [43370.199748] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [43370.199761] Node 0 DMA32: 30695*4kB 87*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 123476kB [43370.199774] Node 0 Normal: 45225*4kB 434*8kB 18*16kB 14*32kB 13*64kB 13*128kB 35*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 196564kB [43370.199788] Node 1 Normal: 25991*4kB 753*8kB 7*16kB 73*32kB 137*64kB 20*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 123764kB [43370.199801] 12733133 total pagecache pages [43370.199803] 0 pages in swap cache [43370.199806] Swap cache stats: add 0, delete 0, find 0/0 [43370.199809] Free swap = 8388604kB [43370.199811] Total swap = 8388604kB [43370.587449] 16777200 pages RAM [43370.587453] 393967 pages reserved [43370.587455] 5843203 pages shared [43370.587457] 10507923 pages non-shared [43370.587507] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [43370.587516] LNetError: 98842:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315258882 total bytes allocated by lnet [43370.587528] LustreError: 98842:0:(niobuf.c:376:ptlrpc_register_bulk()) export-MDT0000-mdc-ffff88080dd47c00: LNetMDAttach failed x1456758881523336/0: rc = -12 [43370.587592] Lustre: 98842:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880c000a7400 x1456758881523336/t0(0) o37->export-MDT0000-mdc-ffff88080dd47c00@172.18.4.5@o2ib:23/10 lens 448/440 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [43370.587627] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection to export-MDT0000 (at 172.18.4.5@o2ib) was lost; in progress operations using this service will wait for recovery to complete [43370.588235] Lustre: export-MDT0000-mdc-ffff88080dd47c00: Connection restored to export-MDT0000 (at 172.18.4.5@o2ib) [71739.716069] nfs: server bias-ib0.local not responding, still trying [71739.716119] nfs: server bias-ib0.local not responding, still trying [71739.716130] nfs: server bias-ib0.local not responding, still trying [71739.716142] nfs: server bias-ib0.local not responding, still trying [71739.716194] nfs: server bias-ib0.local not responding, still trying [71739.775981] nfs: server bias-ib0.local not responding, still trying [71739.823925] nfs: server bias-ib0.local not responding, still trying [71739.831911] nfs: server bias-ib0.local not responding, still trying [71739.887843] nfs: server bias-ib0.local not responding, still trying [71739.915812] nfs: server bias-ib0.local not responding, still trying [71740.027665] nfs: server bias-ib0.local not responding, still trying [71745.101237] nfs: server bias-ib0.local not responding, still trying [71745.101285] nfs: server bias-ib0.local not responding, still trying [71745.101296] nfs: server bias-ib0.local not responding, still trying [71745.101308] nfs: server bias-ib0.local not responding, still trying [71745.101320] nfs: server bias-ib0.local not responding, still trying [71846.308325] nfs: server bias-ib0.local OK [71846.312147] nfs: server bias-ib0.local OK [71846.327383] nfs: server bias-ib0.local OK [71846.328883] nfs: server bias-ib0.local OK [71846.332790] nfs: server bias-ib0.local OK [71846.338157] nfs: server bias-ib0.local OK [71846.339345] nfs: server bias-ib0.local OK [71846.347039] nfs: server bias-ib0.local OK [71846.351262] nfs: server bias-ib0.local OK [71846.369050] nfs: server bias-ib0.local OK [71870.285253] nfs: server bias-ib0.local OK [71995.592167] nfs: server bias-ib0.local OK [71995.850853] nfs: server bias-ib0.local OK [71995.850874] nfs: server bias-ib0.local OK [71995.850880] nfs: server bias-ib0.local OK [71995.850885] nfs: server bias-ib0.local OK [72688.529292] nfs: server bias-ib0.local not responding, still trying [73002.570513] nfs: server bias-ib0.local OK [73188.072024] nfs: server bias-ib0.local not responding, still trying [74037.862939] nfs: server bias-ib0.local OK [74217.727126] nfs: server bias-ib0.local not responding, still trying [74280.311815] nfs: server bias-ib0.local not responding, still trying [74396.504476] nfs: server bias-ib0.local OK [74396.518826] nfs: server bias-ib0.local OK [74659.219773] nfs: server bias-ib0.local not responding, still trying [75158.981293] nfs: server bias-ib0.local OK [79796.038280] The following is only an harmless informational message. [79796.038285] Unless you get a _continuous_flood_ of these messages it means [79796.038286] everything is working fine. Allocations from irqs cannot be [79796.038288] perfectly reliable and the kernel is designed to handle that. [79796.038291] ptlrpcd_2: page allocation failure: order:1, mode:0x40 [79796.038295] Pid: 9217, comm: ptlrpcd_2 Tainted: P NX 3.0.51-0.7.9-default #1 [79796.038297] Call Trace: [79796.038317] [] dump_trace+0x75/0x300 [79796.038325] [] dump_stack+0x69/0x6f [79796.038333] [] warn_alloc_failed+0x102/0x1a0 [79796.038338] [] __alloc_pages_slowpath+0x541/0x7f0 [79796.038343] [] __alloc_pages_nodemask+0x1e9/0x200 [79796.038349] [] kmem_getpages+0x56/0x170 [79796.038354] [] fallback_alloc+0x19b/0x270 [79796.038358] [] __kmalloc+0x285/0x330 [79796.038414] [] ptlrpc_new_bulk+0x45/0x2a0 [ptlrpc] [79796.038478] [] ptlrpc_prep_bulk_imp+0x62/0x1b0 [ptlrpc] [79796.038505] [] osc_brw_prep_request+0x1bf/0xf20 [osc] [79796.038519] [] osc_build_rpc+0x7ba/0x1380 [osc] [79796.038536] [] osc_send_write_rpc+0x1e5/0x470 [osc] [79796.038559] [] osc_check_rpcs+0x5b4/0x710 [osc] [79796.038581] [] osc_io_unplug0+0x35b/0x4c0 [osc] [79796.038601] [] brw_queue_work+0x30/0xd0 [osc] [79796.038629] [] ptlrpc_check_set+0x522/0x1ae0 [ptlrpc] [79796.038677] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [79796.038733] [] ptlrpcd+0x32b/0x410 [ptlrpc] [79796.038766] [] kthread+0x96/0xa0 [79796.038773] [] kernel_thread_helper+0x4/0x10 [79796.038777] Mem-Info: [79796.038778] Node 0 DMA per-cpu: [79796.038781] CPU 0: hi: 0, btch: 1 usd: 0 [79796.038783] CPU 1: hi: 0, btch: 1 usd: 0 [79796.038784] CPU 2: hi: 0, btch: 1 usd: 0 [79796.038786] CPU 3: hi: 0, btch: 1 usd: 0 [79796.038788] CPU 4: hi: 0, btch: 1 usd: 0 [79796.038789] CPU 5: hi: 0, btch: 1 usd: 0 [79796.038791] CPU 6: hi: 0, btch: 1 usd: 0 [79796.038793] CPU 7: hi: 0, btch: 1 usd: 0 [79796.038794] CPU 8: hi: 0, btch: 1 usd: 0 [79796.038796] CPU 9: hi: 0, btch: 1 usd: 0 [79796.038797] CPU 10: hi: 0, btch: 1 usd: 0 [79796.038799] CPU 11: hi: 0, btch: 1 usd: 0 [79796.038801] CPU 12: hi: 0, btch: 1 usd: 0 [79796.038802] CPU 13: hi: 0, btch: 1 usd: 0 [79796.038804] CPU 14: hi: 0, btch: 1 usd: 0 [79796.038805] CPU 15: hi: 0, btch: 1 usd: 0 [79796.038807] CPU 16: hi: 0, btch: 1 usd: 0 [79796.038809] CPU 17: hi: 0, btch: 1 usd: 0 [79796.038810] CPU 18: hi: 0, btch: 1 usd: 0 [79796.038812] CPU 19: hi: 0, btch: 1 usd: 0 [79796.038814] CPU 20: hi: 0, btch: 1 usd: 0 [79796.038815] CPU 21: hi: 0, btch: 1 usd: 0 [79796.038817] CPU 22: hi: 0, btch: 1 usd: 0 [79796.038819] CPU 23: hi: 0, btch: 1 usd: 0 [79796.038820] CPU 24: hi: 0, btch: 1 usd: 0 [79796.038822] CPU 25: hi: 0, btch: 1 usd: 0 [79796.038823] CPU 26: hi: 0, btch: 1 usd: 0 [79796.038825] CPU 27: hi: 0, btch: 1 usd: 0 [79796.038827] CPU 28: hi: 0, btch: 1 usd: 0 [79796.038828] CPU 29: hi: 0, btch: 1 usd: 0 [79796.038830] CPU 30: hi: 0, btch: 1 usd: 0 [79796.038831] CPU 31: hi: 0, btch: 1 usd: 0 [79796.038833] Node 0 DMA32 per-cpu: [79796.038835] CPU 0: hi: 186, btch: 31 usd: 166 [79796.038837] CPU 1: hi: 186, btch: 31 usd: 41 [79796.038838] CPU 2: hi: 186, btch: 31 usd: 97 [79796.038840] CPU 3: hi: 186, btch: 31 usd: 0 [79796.038842] CPU 4: hi: 186, btch: 31 usd: 155 [79796.038843] CPU 5: hi: 186, btch: 31 usd: 22 [79796.038845] CPU 6: hi: 186, btch: 31 usd: 171 [79796.038847] CPU 7: hi: 186, btch: 31 usd: 171 [79796.038848] CPU 8: hi: 186, btch: 31 usd: 12 [79796.038850] CPU 9: hi: 186, btch: 31 usd: 30 [79796.038852] CPU 10: hi: 186, btch: 31 usd: 18 [79796.038853] CPU 11: hi: 186, btch: 31 usd: 60 [79796.038855] CPU 12: hi: 186, btch: 31 usd: 0 [79796.038857] CPU 13: hi: 186, btch: 31 usd: 0 [79796.038858] CPU 14: hi: 186, btch: 31 usd: 0 [79796.038860] CPU 15: hi: 186, btch: 31 usd: 0 [79796.038861] CPU 16: hi: 186, btch: 31 usd: 173 [79796.038863] CPU 17: hi: 186, btch: 31 usd: 173 [79796.038865] CPU 18: hi: 186, btch: 31 usd: 70 [79796.038866] CPU 19: hi: 186, btch: 31 usd: 177 [79796.038868] CPU 20: hi: 186, btch: 31 usd: 157 [79796.038870] CPU 21: hi: 186, btch: 31 usd: 170 [79796.038871] CPU 22: hi: 186, btch: 31 usd: 161 [79796.038873] CPU 23: hi: 186, btch: 31 usd: 0 [79796.038875] CPU 24: hi: 186, btch: 31 usd: 46 [79796.038876] CPU 25: hi: 186, btch: 31 usd: 0 [79796.038878] CPU 26: hi: 186, btch: 31 usd: 0 [79796.038880] CPU 27: hi: 186, btch: 31 usd: 0 [79796.038881] CPU 28: hi: 186, btch: 31 usd: 0 [79796.038883] CPU 29: hi: 186, btch: 31 usd: 0 [79796.038884] CPU 30: hi: 186, btch: 31 usd: 0 [79796.038886] CPU 31: hi: 186, btch: 31 usd: 3 [79796.038887] Node 0 Normal per-cpu: [79796.038889] CPU 0: hi: 186, btch: 31 usd: 174 [79796.038891] CPU 1: hi: 186, btch: 31 usd: 131 [79796.038892] CPU 2: hi: 186, btch: 31 usd: 175 [79796.038894] CPU 3: hi: 186, btch: 31 usd: 16 [79796.038895] CPU 4: hi: 186, btch: 31 usd: 160 [79796.038897] CPU 5: hi: 186, btch: 31 usd: 48 [79796.038899] CPU 6: hi: 186, btch: 31 usd: 182 [79796.038900] CPU 7: hi: 186, btch: 31 usd: 157 [79796.038902] CPU 8: hi: 186, btch: 31 usd: 25 [79796.038903] CPU 9: hi: 186, btch: 31 usd: 30 [79796.038905] CPU 10: hi: 186, btch: 31 usd: 133 [79796.038907] CPU 11: hi: 186, btch: 31 usd: 148 [79796.038908] CPU 12: hi: 186, btch: 31 usd: 0 [79796.038910] CPU 13: hi: 186, btch: 31 usd: 0 [79796.038911] CPU 14: hi: 186, btch: 31 usd: 0 [79796.038913] CPU 15: hi: 186, btch: 31 usd: 30 [79796.038914] CPU 16: hi: 186, btch: 31 usd: 183 [79796.038916] CPU 17: hi: 186, btch: 31 usd: 125 [79796.038917] CPU 18: hi: 186, btch: 31 usd: 64 [79796.038919] CPU 19: hi: 186, btch: 31 usd: 35 [79796.038920] CPU 20: hi: 186, btch: 31 usd: 185 [79796.038922] CPU 21: hi: 186, btch: 31 usd: 98 [79796.038924] CPU 22: hi: 186, btch: 31 usd: 184 [79796.038925] CPU 23: hi: 186, btch: 31 usd: 93 [79796.038927] CPU 24: hi: 186, btch: 31 usd: 13 [79796.038928] CPU 25: hi: 186, btch: 31 usd: 25 [79796.038930] CPU 26: hi: 186, btch: 31 usd: 0 [79796.038931] CPU 27: hi: 186, btch: 31 usd: 22 [79796.038933] CPU 28: hi: 186, btch: 31 usd: 0 [79796.038934] CPU 29: hi: 186, btch: 31 usd: 0 [79796.038936] CPU 30: hi: 186, btch: 31 usd: 0 [79796.038937] CPU 31: hi: 186, btch: 31 usd: 6 [79796.038939] Node 1 Normal per-cpu: [79796.038941] CPU 0: hi: 186, btch: 31 usd: 0 [79796.038942] CPU 1: hi: 186, btch: 31 usd: 0 [79796.038944] CPU 2: hi: 186, btch: 31 usd: 6 [79796.038946] CPU 3: hi: 186, btch: 31 usd: 158 [79796.038947] CPU 4: hi: 186, btch: 31 usd: 71 [79796.038949] CPU 5: hi: 186, btch: 31 usd: 119 [79796.038951] CPU 6: hi: 186, btch: 31 usd: 72 [79796.038952] CPU 7: hi: 186, btch: 31 usd: 134 [79796.038954] CPU 8: hi: 186, btch: 31 usd: 29 [79796.038956] CPU 9: hi: 186, btch: 31 usd: 196 [79796.038957] CPU 10: hi: 186, btch: 31 usd: 175 [79796.038959] CPU 11: hi: 186, btch: 31 usd: 162 [79796.038960] CPU 12: hi: 186, btch: 31 usd: 1 [79796.038962] CPU 13: hi: 186, btch: 31 usd: 181 [79796.038964] CPU 14: hi: 186, btch: 31 usd: 180 [79796.038965] CPU 15: hi: 186, btch: 31 usd: 164 [79796.038967] CPU 16: hi: 186, btch: 31 usd: 7 [79796.038969] CPU 17: hi: 186, btch: 31 usd: 118 [79796.038970] CPU 18: hi: 186, btch: 31 usd: 0 [79796.038972] CPU 19: hi: 186, btch: 31 usd: 0 [79796.038973] CPU 20: hi: 186, btch: 31 usd: 0 [79796.038975] CPU 21: hi: 186, btch: 31 usd: 0 [79796.038977] CPU 22: hi: 186, btch: 31 usd: 73 [79796.038979] CPU 23: hi: 186, btch: 31 usd: 0 [79796.038980] CPU 24: hi: 186, btch: 31 usd: 36 [79796.038982] CPU 25: hi: 186, btch: 31 usd: 182 [79796.038984] CPU 26: hi: 186, btch: 31 usd: 160 [79796.038985] CPU 27: hi: 186, btch: 31 usd: 159 [79796.038987] CPU 28: hi: 186, btch: 31 usd: 169 [79796.038988] CPU 29: hi: 186, btch: 31 usd: 159 [79796.038990] CPU 30: hi: 186, btch: 31 usd: 199 [79796.038992] CPU 31: hi: 186, btch: 31 usd: 165 [79796.038996] active_anon:193049 inactive_anon:48914 isolated_anon:0 [79796.038997] active_file:49738 inactive_file:14903160 isolated_file:32 [79796.038998] unevictable:2648 dirty:51913 writeback:385 unstable:24895 [79796.038999] free:66580 slab_reclaimable:302612 slab_unreclaimable:444787 [79796.039000] mapped:4624 shmem:285 pagetables:3158 bounce:0 [79796.039002] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [79796.039011] lowmem_reserve[]: 0 2959 32249 32249 [79796.039014] Node 0 DMA32 free:123516kB min:4128kB low:5160kB high:6192kB active_anon:7976kB inactive_anon:3080kB active_file:6436kB inactive_file:1904032kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:3364kB writeback:104kB mapped:356kB shmem:0kB slab_reclaimable:353456kB slab_unreclaimable:68404kB kernel_stack:72kB pagetables:192kB unstable:2484kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [79796.039023] lowmem_reserve[]: 0 0 29290 29290 [79796.039026] Node 0 Normal free:61396kB min:40864kB low:51080kB high:61296kB active_anon:573000kB inactive_anon:105676kB active_file:142012kB inactive_file:27294492kB unevictable:8528kB isolated(anon):0kB isolated(file):128kB present:29992960kB mlocked:2396kB dirty:161996kB writeback:0kB mapped:10900kB shmem:244kB slab_reclaimable:530444kB slab_unreclaimable:263344kB kernel_stack:2408kB pagetables:7308kB unstable:7684kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [79796.039036] lowmem_reserve[]: 0 0 0 0 [79796.039038] Node 1 Normal free:65936kB min:45092kB low:56364kB high:67636kB active_anon:191220kB inactive_anon:86900kB active_file:50504kB inactive_file:30413732kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:42292kB writeback:1508kB mapped:7240kB shmem:896kB slab_reclaimable:326548kB slab_unreclaimable:1447400kB kernel_stack:3064kB pagetables:5132kB unstable:89412kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [79796.039048] lowmem_reserve[]: 0 0 0 0 [79796.039050] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [79796.039057] Node 0 DMA32: 30658*4kB 43*8kB 4*16kB 2*32kB 2*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 123872kB [79796.039064] Node 0 Normal: 13573*4kB 117*8kB 74*16kB 52*32kB 46*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61148kB [79796.039071] Node 1 Normal: 15651*4kB 270*8kB 31*16kB 19*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 66060kB [79796.039078] 14953804 total pagecache pages [79796.039080] 5 pages in swap cache [79796.039081] Swap cache stats: add 48, delete 43, find 0/0 [79796.039083] Free swap = 8388412kB [79796.039084] Total swap = 8388604kB [79796.267352] 16777200 pages RAM [79796.267355] 393967 pages reserved [79796.267356] 7572065 pages shared [79796.267357] 8757262 pages non-shared [79796.267403] LustreError: 9217:0:(osc_request.c:2198:osc_build_rpc()) prep_req failed: -12 [79796.267751] LustreError: 9217:0:(osc_cache.c:2211:osc_check_rpcs()) Write request failed with -12 [80721.560896] The following is only an harmless informational message. [80721.560899] Unless you get a _continuous_flood_ of these messages it means [80721.560901] everything is working fine. Allocations from irqs cannot be [80721.560902] perfectly reliable and the kernel is designed to handle that. [80721.560905] ptlrpcd_14: page allocation failure: order:1, mode:0x40 [80721.560908] Pid: 9229, comm: ptlrpcd_14 Tainted: P NX 3.0.51-0.7.9-default #1 [80721.560911] Call Trace: [80721.560930] [] dump_trace+0x75/0x300 [80721.560939] [] dump_stack+0x69/0x6f [80721.560946] [] warn_alloc_failed+0x102/0x1a0 [80721.560951] [] __alloc_pages_slowpath+0x541/0x7f0 [80721.560956] [] __alloc_pages_nodemask+0x1e9/0x200 [80721.560962] [] kmem_getpages+0x56/0x170 [80721.560967] [] fallback_alloc+0x19b/0x270 [80721.560971] [] __kmalloc+0x285/0x330 [80721.560992] [] LNetMDAttach+0x163/0x5b0 [lnet] [80721.561070] [] ptlrpc_register_bulk+0x258/0x9e0 [ptlrpc] [80721.561115] [] ptl_send_rpc+0x174/0xc70 [ptlrpc] [80721.561158] [] ptlrpc_send_new_req+0x4a0/0x8a0 [ptlrpc] [80721.561198] [] ptlrpc_check_set+0x408/0x1ae0 [ptlrpc] [80721.561245] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [80721.561301] [] ptlrpcd+0x27b/0x410 [ptlrpc] [80721.561334] [] kthread+0x96/0xa0 [80721.561341] [] kernel_thread_helper+0x4/0x10 [80721.561345] Mem-Info: [80721.561347] Node 0 DMA per-cpu: [80721.561349] CPU 0: hi: 0, btch: 1 usd: 0 [80721.561351] CPU 1: hi: 0, btch: 1 usd: 0 [80721.561353] CPU 2: hi: 0, btch: 1 usd: 0 [80721.561354] CPU 3: hi: 0, btch: 1 usd: 0 [80721.561356] CPU 4: hi: 0, btch: 1 usd: 0 [80721.561358] CPU 5: hi: 0, btch: 1 usd: 0 [80721.561360] CPU 6: hi: 0, btch: 1 usd: 0 [80721.561361] CPU 7: hi: 0, btch: 1 usd: 0 [80721.561363] CPU 8: hi: 0, btch: 1 usd: 0 [80721.561364] CPU 9: hi: 0, btch: 1 usd: 0 [80721.561366] CPU 10: hi: 0, btch: 1 usd: 0 [80721.561368] CPU 11: hi: 0, btch: 1 usd: 0 [80721.561369] CPU 12: hi: 0, btch: 1 usd: 0 [80721.561371] CPU 13: hi: 0, btch: 1 usd: 0 [80721.561372] CPU 14: hi: 0, btch: 1 usd: 0 [80721.561374] CPU 15: hi: 0, btch: 1 usd: 0 [80721.561376] CPU 16: hi: 0, btch: 1 usd: 0 [80721.561377] CPU 17: hi: 0, btch: 1 usd: 0 [80721.561379] CPU 18: hi: 0, btch: 1 usd: 0 [80721.561381] CPU 19: hi: 0, btch: 1 usd: 0 [80721.561382] CPU 20: hi: 0, btch: 1 usd: 0 [80721.561384] CPU 21: hi: 0, btch: 1 usd: 0 [80721.561386] CPU 22: hi: 0, btch: 1 usd: 0 [80721.561387] CPU 23: hi: 0, btch: 1 usd: 0 [80721.561389] CPU 24: hi: 0, btch: 1 usd: 0 [80721.561390] CPU 25: hi: 0, btch: 1 usd: 0 [80721.561392] CPU 26: hi: 0, btch: 1 usd: 0 [80721.561394] CPU 27: hi: 0, btch: 1 usd: 0 [80721.561395] CPU 28: hi: 0, btch: 1 usd: 0 [80721.561397] CPU 29: hi: 0, btch: 1 usd: 0 [80721.561398] CPU 30: hi: 0, btch: 1 usd: 0 [80721.561400] CPU 31: hi: 0, btch: 1 usd: 0 [80721.561401] Node 0 DMA32 per-cpu: [80721.561404] CPU 0: hi: 186, btch: 31 usd: 45 [80721.561405] CPU 1: hi: 186, btch: 31 usd: 31 [80721.561407] CPU 2: hi: 186, btch: 31 usd: 199 [80721.561409] CPU 3: hi: 186, btch: 31 usd: 89 [80721.561410] CPU 4: hi: 186, btch: 31 usd: 0 [80721.561412] CPU 5: hi: 186, btch: 31 usd: 157 [80721.561414] CPU 6: hi: 186, btch: 31 usd: 42 [80721.561416] CPU 7: hi: 186, btch: 31 usd: 173 [80721.561417] CPU 8: hi: 186, btch: 31 usd: 4 [80721.561419] CPU 9: hi: 186, btch: 31 usd: 28 [80721.561420] CPU 10: hi: 186, btch: 31 usd: 179 [80721.561422] CPU 11: hi: 186, btch: 31 usd: 174 [80721.561424] CPU 12: hi: 186, btch: 31 usd: 18 [80721.561425] CPU 13: hi: 186, btch: 31 usd: 79 [80721.561427] CPU 14: hi: 186, btch: 31 usd: 157 [80721.561429] CPU 15: hi: 186, btch: 31 usd: 91 [80721.561430] CPU 16: hi: 186, btch: 31 usd: 34 [80721.561432] CPU 17: hi: 186, btch: 31 usd: 39 [80721.561434] CPU 18: hi: 186, btch: 31 usd: 34 [80721.561435] CPU 19: hi: 186, btch: 31 usd: 4 [80721.561437] CPU 20: hi: 186, btch: 31 usd: 161 [80721.561439] CPU 21: hi: 186, btch: 31 usd: 176 [80721.561441] CPU 22: hi: 186, btch: 31 usd: 166 [80721.561442] CPU 23: hi: 186, btch: 31 usd: 204 [80721.561444] CPU 24: hi: 186, btch: 31 usd: 107 [80721.561446] CPU 25: hi: 186, btch: 31 usd: 114 [80721.561447] CPU 26: hi: 186, btch: 31 usd: 0 [80721.561449] CPU 27: hi: 186, btch: 31 usd: 86 [80721.561451] CPU 28: hi: 186, btch: 31 usd: 0 [80721.561454] CPU 29: hi: 186, btch: 31 usd: 0 [80721.561456] CPU 30: hi: 186, btch: 31 usd: 147 [80721.561459] CPU 31: hi: 186, btch: 31 usd: 181 [80721.561460] Node 0 Normal per-cpu: [80721.561463] CPU 0: hi: 186, btch: 31 usd: 162 [80721.561465] CPU 1: hi: 186, btch: 31 usd: 146 [80721.561468] CPU 2: hi: 186, btch: 31 usd: 31 [80721.561469] CPU 3: hi: 186, btch: 31 usd: 56 [80721.561471] CPU 4: hi: 186, btch: 31 usd: 146 [80721.561473] CPU 5: hi: 186, btch: 31 usd: 149 [80721.561474] CPU 6: hi: 186, btch: 31 usd: 64 [80721.561476] CPU 7: hi: 186, btch: 31 usd: 178 [80721.561477] CPU 8: hi: 186, btch: 31 usd: 0 [80721.561479] CPU 9: hi: 186, btch: 31 usd: 52 [80721.561480] CPU 10: hi: 186, btch: 31 usd: 175 [80721.561482] CPU 11: hi: 186, btch: 31 usd: 181 [80721.561483] CPU 12: hi: 186, btch: 31 usd: 138 [80721.561485] CPU 13: hi: 186, btch: 31 usd: 176 [80721.561487] CPU 14: hi: 186, btch: 31 usd: 142 [80721.561488] CPU 15: hi: 186, btch: 31 usd: 139 [80721.561490] CPU 16: hi: 186, btch: 31 usd: 174 [80721.561492] CPU 17: hi: 186, btch: 31 usd: 81 [80721.561493] CPU 18: hi: 186, btch: 31 usd: 42 [80721.561495] CPU 19: hi: 186, btch: 31 usd: 56 [80721.561496] CPU 20: hi: 186, btch: 31 usd: 32 [80721.561498] CPU 21: hi: 186, btch: 31 usd: 187 [80721.561500] CPU 22: hi: 186, btch: 31 usd: 160 [80721.561501] CPU 23: hi: 186, btch: 31 usd: 106 [80721.561503] CPU 24: hi: 186, btch: 31 usd: 175 [80721.561504] CPU 25: hi: 186, btch: 31 usd: 133 [80721.561506] CPU 26: hi: 186, btch: 31 usd: 137 [80721.561507] CPU 27: hi: 186, btch: 31 usd: 173 [80721.561509] CPU 28: hi: 186, btch: 31 usd: 0 [80721.561510] CPU 29: hi: 186, btch: 31 usd: 0 [80721.561512] CPU 30: hi: 186, btch: 31 usd: 169 [80721.561513] CPU 31: hi: 186, btch: 31 usd: 179 [80721.561515] Node 1 Normal per-cpu: [80721.561517] CPU 0: hi: 186, btch: 31 usd: 0 [80721.561518] CPU 1: hi: 186, btch: 31 usd: 0 [80721.561520] CPU 2: hi: 186, btch: 31 usd: 0 [80721.561522] CPU 3: hi: 186, btch: 31 usd: 17 [80721.561523] CPU 4: hi: 186, btch: 31 usd: 0 [80721.561525] CPU 5: hi: 186, btch: 31 usd: 0 [80721.561527] CPU 6: hi: 186, btch: 31 usd: 131 [80721.561529] CPU 7: hi: 186, btch: 31 usd: 0 [80721.561530] CPU 8: hi: 186, btch: 31 usd: 61 [80721.561532] CPU 9: hi: 186, btch: 31 usd: 45 [80721.561533] CPU 10: hi: 186, btch: 31 usd: 170 [80721.561535] CPU 11: hi: 186, btch: 31 usd: 162 [80721.561536] CPU 12: hi: 186, btch: 31 usd: 162 [80721.561538] CPU 13: hi: 186, btch: 31 usd: 161 [80721.561540] CPU 14: hi: 186, btch: 31 usd: 168 [80721.561541] CPU 15: hi: 186, btch: 31 usd: 167 [80721.561543] CPU 16: hi: 186, btch: 31 usd: 0 [80721.561545] CPU 17: hi: 186, btch: 31 usd: 11 [80721.561546] CPU 18: hi: 186, btch: 31 usd: 15 [80721.561548] CPU 19: hi: 186, btch: 31 usd: 0 [80721.561550] CPU 20: hi: 186, btch: 31 usd: 0 [80721.561551] CPU 21: hi: 186, btch: 31 usd: 0 [80721.561553] CPU 22: hi: 186, btch: 31 usd: 17 [80721.561555] CPU 23: hi: 186, btch: 31 usd: 0 [80721.561556] CPU 24: hi: 186, btch: 31 usd: 160 [80721.561558] CPU 25: hi: 186, btch: 31 usd: 162 [80721.561560] CPU 26: hi: 186, btch: 31 usd: 116 [80721.561561] CPU 27: hi: 186, btch: 31 usd: 179 [80721.561563] CPU 28: hi: 186, btch: 31 usd: 157 [80721.561565] CPU 29: hi: 186, btch: 31 usd: 74 [80721.561566] CPU 30: hi: 186, btch: 31 usd: 184 [80721.561568] CPU 31: hi: 186, btch: 31 usd: 158 [80721.561573] active_anon:193048 inactive_anon:48914 isolated_anon:0 [80721.561573] active_file:55369 inactive_file:14616020 isolated_file:0 [80721.561574] unevictable:2648 dirty:51066 writeback:0 unstable:22339 [80721.561575] free:63734 slab_reclaimable:227499 slab_unreclaimable:797776 [80721.561576] mapped:4881 shmem:285 pagetables:3129 bounce:0 [80721.561578] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [80721.561587] lowmem_reserve[]: 0 2959 32249 32249 [80721.561590] Node 0 DMA32 free:124060kB min:4128kB low:5160kB high:6192kB active_anon:8472kB inactive_anon:3080kB active_file:7128kB inactive_file:1897488kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:5416kB writeback:876kB mapped:356kB shmem:0kB slab_reclaimable:325404kB slab_unreclaimable:98196kB kernel_stack:88kB pagetables:276kB unstable:1448kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [80721.561600] lowmem_reserve[]: 0 0 29290 29290 [80721.561602] Node 0 Normal free:55908kB min:40864kB low:51080kB high:61296kB active_anon:572536kB inactive_anon:105676kB active_file:146792kB inactive_file:27084428kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:191380kB writeback:0kB mapped:10912kB shmem:244kB slab_reclaimable:446572kB slab_unreclaimable:534332kB kernel_stack:2400kB pagetables:7108kB unstable:8812kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [80721.561612] lowmem_reserve[]: 0 0 0 0 [80721.561614] Node 1 Normal free:59496kB min:45092kB low:56364kB high:67636kB active_anon:191184kB inactive_anon:86900kB active_file:67556kB inactive_file:29481776kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:33095680kB mlocked:2064kB dirty:7468kB writeback:0kB mapped:8256kB shmem:896kB slab_reclaimable:138020kB slab_unreclaimable:2558576kB kernel_stack:3048kB pagetables:5132kB unstable:79096kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no [80721.561623] lowmem_reserve[]: 0 0 0 0 [80721.561626] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [80721.561633] Node 0 DMA32: 30555*4kB 48*8kB 3*16kB 2*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 124124kB [80721.561640] Node 0 Normal: 10273*4kB 147*8kB 4*16kB 55*32kB 153*64kB 11*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 55292kB [80721.561647] Node 1 Normal: 14110*4kB 323*8kB 25*16kB 6*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59744kB [80721.561654] 14672470 total pagecache pages [80721.561658] 5 pages in swap cache [80721.561660] Swap cache stats: add 48, delete 43, find 0/0 [80721.561661] Free swap = 8388412kB [80721.561662] Total swap = 8388604kB [80721.778517] 16777200 pages RAM [80721.778520] 393967 pages reserved [80721.778521] 5801906 pages shared [80721.778522] 10528373 pages non-shared [80721.778547] LNetError: 9229:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: out of memory at /usr/src/packages/BUILD/lustre-2.5.52/lnet/include/lnet/lib-lnet.h:468 (tried to alloc '(md)' = 4208) [80721.778552] LNetError: 9229:0:(lib-lnet.h:468:lnet_md_alloc()) LNET: 315259106 total bytes allocated by lnet [80721.778559] LustreError: 9229:0:(niobuf.c:376:ptlrpc_register_bulk()) export-OST0008-osc-ffff88080dd47c00: LNetMDAttach failed x1456758990867504/0: rc = -12 [80721.778587] Lustre: 9229:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880dc4f87400 x1456758990867504/t0(0) o4->export-OST0008-osc-ffff88080dd47c00@172.18.4.8@o2ib:6/4 lens 488/448 e 0 to 1 dl 0 ref 2 fl Rpc:X/0/ffffffff rc -12/-1 [80721.778599] Lustre: export-OST0008-osc-ffff88080dd47c00: Connection to export-OST0008 (at 172.18.4.8@o2ib) was lost; in progress operations using this service will wait for recovery to complete [80721.789651] Lustre: export-OST0008-osc-ffff88080dd47c00: Connection restored to export-OST0008 (at 172.18.4.8@o2ib) [98227.193289] megasas: Found FW in FAULT state, will reset adapter. [98227.193294] megaraid_sas: resetting fusion adapter. [98235.287020] megasas: Waiting for FW to come to ready state [98320.834547] megasas: FW now in Ready state [98321.589603] megasas:IOC Init cmd success [98321.613589] megaraid_sas: Reset successful. [100877.837896] megasas: Found FW in FAULT state, will reset adapter. [100877.837902] megaraid_sas: resetting fusion adapter. [100885.927639] megasas: Waiting for FW to come to ready state [100971.998552] megasas: FW now in Ready state [100973.356907] megasas:IOC Init cmd success [100973.380918] megaraid_sas: Reset successful. [101035.098601] megasas: Found FW in FAULT state, will reset adapter. [101035.098605] megaraid_sas: resetting fusion adapter. [101043.188419] megasas: Waiting for FW to come to ready state [101128.783919] megasas: FW now in Ready state [101129.574928] megasas:IOC Init cmd success [101129.598946] megaraid_sas: Reset successful. [101312.203358] megasas: [ 0]waiting for 51 commands to complete [101312.203546] megasas: waitingfor controller reset to finish [101312.203567] megasas: waitingfor controller reset to finish [103593.395366] megasas: Found FW in FAULT state, will reset adapter. [103593.395373] megaraid_sas: resetting fusion adapter. [103601.493163] megasas: Waiting for FW to come to ready state [103686.769035] megasas: FW now in Ready state [103687.895612] megasas:IOC Init cmd success [103687.919517] megaraid_sas: Reset successful. [103869.984683] megasas: [ 0]waiting for 8 commands to complete [103869.984945] megasas: waitingfor controller reset to finish [103869.984964] megasas: waitingfor controller reset to finish [104093.449347] megasas: Found FW in FAULT state, will reset adapter. [104093.449351] megaraid_sas: resetting fusion adapter. [104101.539213] megasas: Waiting for FW to come to ready state [104186.926987] megasas: FW now in Ready state [104188.029560] megasas:IOC Init cmd success [104188.053480] megaraid_sas: Reset successful. [104655.966437] The following is only an harmless informational message. [104655.966446] Unless you get a _continuous_flood_ of these messages it means [104655.966451] everything is working fine. Allocations from irqs cannot be [104655.966455] perfectly reliable and the kernel is designed to handle that. [104655.966462] ptlrpcd_4: page allocation failure: order:1, mode:0x40 [104655.966471] Pid: 9219, comm: ptlrpcd_4 Tainted: P NX 3.0.51-0.7.9-default #1 [104655.966476] Call Trace: [104655.966512] [] dump_trace+0x75/0x300 [104655.966530] [] dump_stack+0x69/0x6f [104655.966546] [] warn_alloc_failed+0x102/0x1a0 [104655.966560] [] __alloc_pages_slowpath+0x541/0x7f0 [104655.966573] [] __alloc_pages_nodemask+0x1e9/0x200 [104655.966587] [] kmem_getpages+0x56/0x170 [104655.966599] [] fallback_alloc+0x19b/0x270 [104655.966611] [] __kmalloc+0x285/0x330 [104655.966714] [] ptlrpc_new_bulk+0x45/0x2a0 [ptlrpc] [104655.966837] [] ptlrpc_prep_bulk_imp+0x62/0x1b0 [ptlrpc] [104655.966886] [] osc_brw_prep_request+0x1bf/0xf20 [osc] [104655.966912] [] osc_build_rpc+0x7ba/0x1380 [osc] [104655.966941] [] osc_send_write_rpc+0x1e5/0x470 [osc] [104655.966983] [] osc_check_rpcs+0x5b4/0x710 [osc] [104655.967025] [] osc_io_unplug0+0x35b/0x4c0 [osc] [104655.967061] [] brw_queue_work+0x30/0xd0 [osc] [104655.967113] [] ptlrpc_check_set+0x522/0x1ae0 [ptlrpc] [104655.967198] [] ptlrpcd_check+0x52b/0x550 [ptlrpc] [104655.967302] [] ptlrpcd+0x27b/0x410 [ptlrpc] [104655.967361] [] kthread+0x96/0xa0 [104655.967372] [] kernel_thread_helper+0x4/0x10 [104655.967379] Mem-Info: [104655.967382] Node 0 DMA per-cpu: [104655.967386] CPU 0: hi: 0, btch: 1 usd: 0 [104655.967389] CPU 1: hi: 0, btch: 1 usd: 0 [104655.967392] CPU 2: hi: 0, btch: 1 usd: 0 [104655.967395] CPU 3: hi: 0, btch: 1 usd: 0 [104655.967398] CPU 4: hi: 0, btch: 1 usd: 0 [104655.967401] CPU 5: hi: 0, btch: 1 usd: 0 [104655.967404] CPU 6: hi: 0, btch: 1 usd: 0 [104655.967407] CPU 7: hi: 0, btch: 1 usd: 0 [104655.967410] CPU 8: hi: 0, btch: 1 usd: 0 [104655.967413] CPU 9: hi: 0, btch: 1 usd: 0 [104655.967416] CPU 10: hi: 0, btch: 1 usd: 0 [104655.967419] CPU 11: hi: 0, btch: 1 usd: 0 [104655.967422] CPU 12: hi: 0, btch: 1 usd: 0 [104655.967425] CPU 13: hi: 0, btch: 1 usd: 0 [104655.967428] CPU 14: hi: 0, btch: 1 usd: 0 [104655.967431] CPU 15: hi: 0, btch: 1 usd: 0 [104655.967434] CPU 16: hi: 0, btch: 1 usd: 0 [104655.967437] CPU 17: hi: 0, btch: 1 usd: 0 [104655.967439] CPU 18: hi: 0, btch: 1 usd: 0 [104655.967443] CPU 19: hi: 0, btch: 1 usd: 0 [104655.967446] CPU 20: hi: 0, btch: 1 usd: 0 [104655.967449] CPU 21: hi: 0, btch: 1 usd: 0 [104655.967451] CPU 22: hi: 0, btch: 1 usd: 0 [104655.967454] CPU 23: hi: 0, btch: 1 usd: 0 [104655.967457] CPU 24: hi: 0, btch: 1 usd: 0 [104655.967460] CPU 25: hi: 0, btch: 1 usd: 0 [104655.967463] CPU 26: hi: 0, btch: 1 usd: 0 [104655.967466] CPU 27: hi: 0, btch: 1 usd: 0 [104655.967469] CPU 28: hi: 0, btch: 1 usd: 0 [104655.967472] CPU 29: hi: 0, btch: 1 usd: 0 [104655.967475] CPU 30: hi: 0, btch: 1 usd: 0 [104655.967477] CPU 31: hi: 0, btch: 1 usd: 0 [104655.967480] Node 0 DMA32 per-cpu: [104655.967484] CPU 0: hi: 186, btch: 31 usd: 30 [104655.967487] CPU 1: hi: 186, btch: 31 usd: 45 [104655.967491] CPU 2: hi: 186, btch: 31 usd: 104 [104655.967494] CPU 3: hi: 186, btch: 31 usd: 89 [104655.967497] CPU 4: hi: 186, btch: 31 usd: 182 [104655.967500] CPU 5: hi: 186, btch: 31 usd: 159 [104655.967503] CPU 6: hi: 186, btch: 31 usd: 155 [104655.967506] CPU 7: hi: 186, btch: 31 usd: 59 [104655.967509] CPU 8: hi: 186, btch: 31 usd: 0 [104655.967512] CPU 9: hi: 186, btch: 31 usd: 0 [104655.967515] CPU 10: hi: 186, btch: 31 usd: 0 [104655.967518] CPU 11: hi: 186, btch: 31 usd: 12 [104655.967521] CPU 12: hi: 186, btch: 31 usd: 21 [104655.967524] CPU 13: hi: 186, btch: 31 usd: 30 [104655.967527] CPU 14: hi: 186, btch: 31 usd: 0 [104655.967530] CPU 15: hi: 186, btch: 31 usd: 0 [104655.967533] CPU 16: hi: 186, btch: 31 usd: 167 [104655.967536] CPU 17: hi: 186, btch: 31 usd: 181 [104655.967539] CPU 18: hi: 186, btch: 31 usd: 145 [104655.967542] CPU 19: hi: 186, btch: 31 usd: 166 [104655.967545] CPU 20: hi: 186, btch: 31 usd: 181 [104655.967548] CPU 21: hi: 186, btch: 31 usd: 161 [104655.967551] CPU 22: hi: 186, btch: 31 usd: 192 [104655.967554] CPU 23: hi: 186, btch: 31 usd: 0 [104655.967557] CPU 24: hi: 186, btch: 31 usd: 18 [104655.967560] CPU 25: hi: 186, btch: 31 usd: 0 [104655.967563] CPU 26: hi: 186, btch: 31 usd: 87 [104655.967566] CPU 27: hi: 186, btch: 31 usd: 0 [104655.967569] CPU 28: hi: 186, btch: 31 usd: 0 [104655.967572] CPU 29: hi: 186, btch: 31 usd: 0 [104655.967575] CPU 30: hi: 186, btch: 31 usd: 1 [104655.967578] CPU 31: hi: 186, btch: 31 usd: 0 [104655.967580] Node 0 Normal per-cpu: [104655.967584] CPU 0: hi: 186, btch: 31 usd: 59 [104655.967587] CPU 1: hi: 186, btch: 31 usd: 142 [104655.967590] CPU 2: hi: 186, btch: 31 usd: 171 [104655.967593] CPU 3: hi: 186, btch: 31 usd: 53 [104655.967595] CPU 4: hi: 186, btch: 31 usd: 110 [104655.967598] CPU 5: hi: 186, btch: 31 usd: 160 [104655.967601] CPU 6: hi: 186, btch: 31 usd: 163 [104655.967604] CPU 7: hi: 186, btch: 31 usd: 23 [104655.967607] CPU 8: hi: 186, btch: 31 usd: 0 [104655.967610] CPU 9: hi: 186, btch: 31 usd: 0 [104655.967613] CPU 10: hi: 186, btch: 31 usd: 130 [104655.967616] CPU 11: hi: 186, btch: 31 usd: 164 [104655.967619] CPU 12: hi: 186, btch: 31 usd: 30 [104655.967621] CPU 13: hi: 186, btch: 31 usd: 5 [104655.967624] CPU 14: hi: 186, btch: 31 usd: 0 [104655.967627] CPU 15: hi: 186, btch: 31 usd: 0 [104655.967630] CPU 16: hi: 186, btch: 31 usd: 81 [104655.967633] CPU 17: hi: 186, btch: 31 usd: 109 [104655.967636] CPU 18: hi: 186, btch: 31 usd: 140 [104655.967639] CPU 19: hi: 186, btch: 31 usd: 173 [104655.967641] CPU 20: hi: 186, btch: 31 usd: 157 [104655.967644] CPU 21: hi: 186, btch: 31 usd: 155 [104655.967648] CPU 22: hi: 186, btch: 31 usd: 75 [104655.967651] CPU 23: hi: 186, btch: 31 usd: 49 [104655.967653] CPU 24: hi: 186, btch: 31 usd: 1 [104655.967656] CPU 25: hi: 186, btch: 31 usd: 106 [104655.967659] CPU 26: hi: 186, btch: 31 usd: 23 [104655.967662] CPU 27: hi: 186, btch: 31 usd: 0 [104655.967665] CPU 28: hi: 186, btch: 31 usd: 0 [104655.967668] CPU 29: hi: 186, btch: 31 usd: 0 [104655.967671] CPU 30: hi: 186, btch: 31 usd: 0 [104655.967673] CPU 31: hi: 186, btch: 31 usd: 5 [104655.967676] Node 1 Normal per-cpu: [104655.967680] CPU 0: hi: 186, btch: 31 usd: 31 [104655.967683] CPU 1: hi: 186, btch: 31 usd: 0 [104655.967686] CPU 2: hi: 186, btch: 31 usd: 113 [104655.967689] CPU 3: hi: 186, btch: 31 usd: 14 [104655.967692] CPU 4: hi: 186, btch: 31 usd: 0 [104655.967695] CPU 5: hi: 186, btch: 31 usd: 0 [104655.967698] CPU 6: hi: 186, btch: 31 usd: 145 [104655.967701] CPU 7: hi: 186, btch: 31 usd: 38 [104655.967704] CPU 8: hi: 186, btch: 31 usd: 167 [104655.967707] CPU 9: hi: 186, btch: 31 usd: 99 [104655.967710] CPU 10: hi: 186, btch: 31 usd: 171 [104655.967713] CPU 11: hi: 186, btch: 31 usd: 182 [104655.967716] CPU 12: hi: 186, btch: 31 usd: 76 [104655.967719] CPU 13: hi: 186, btch: 31 usd: 202 [104655.967722] CPU 14: hi: 186, btch: 31 usd: 170 [104655.967725] CPU 15: hi: 186, btch: 31 usd: 125 [104655.967728] CPU 16: hi: 186, btch: 31 usd: 0 [104655.967731] CPU 17: hi: 186, btch: 31 usd: 7 [104655.967734] CPU 18: hi: 186, btch: 31 usd: 89 [104655.967737] CPU 19: hi: 186, btch: 31 usd: 0 [104655.967740] CPU 20: hi: 186, btch: 31 usd: 0 [104655.967743] CPU 21: hi: 186, btch: 31 usd: 0 [104655.967746] CPU 22: hi: 186, btch: 31 usd: 31 [104655.967749] CPU 23: hi: 186, btch: 31 usd: 178 [104655.967752] CPU 24: hi: 186, btch: 31 usd: 149 [104655.967755] CPU 25: hi: 186, btch: 31 usd: 164 [104655.967758] CPU 26: hi: 186, btch: 31 usd: 160 [104655.967761] CPU 27: hi: 186, btch: 31 usd: 181 [104655.967763] CPU 28: hi: 186, btch: 31 usd: 68 [104655.967766] CPU 29: hi: 186, btch: 31 usd: 39 [104655.967769] CPU 30: hi: 186, btch: 31 usd: 40 [104655.967772] CPU 31: hi: 186, btch: 31 usd: 134 [104655.967780] active_anon:345551 inactive_anon:49932 isolated_anon:0 [104655.967782] active_file:50628 inactive_file:14031093 isolated_file:32 [104655.967784] unevictable:2648 dirty:187844 writeback:307 unstable:12894 [104655.967785] free:70344 slab_reclaimable:900202 slab_unreclaimable:448160 [104655.967787] mapped:5904 shmem:285 pagetables:3709 bounce:0 [104655.967791] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15620kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [104655.967807] lowmem_reserve[]: 0 2959 32249 32249 [104655.967812] Node 0 DMA32 free:123916kB min:4128kB low:5160kB high:6192kB active_anon:12356kB inactive_anon:7060kB active_file:892kB inactive_file:1897712kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3030348kB mlocked:0kB dirty:25080kB writeback:136kB mapped:224kB shmem:0kB slab_reclaimable:349820kB slab_unreclaimable:64360kB kernel_stack:96kB pagetables:196kB unstable:1160kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [104655.967829] lowmem_reserve[]: 0 0 29290 29290 [104655.967834] Node 0 Normal free:68068kB min:40864kB low:51080kB high:61296kB active_anon:962748kB inactive_anon:105760kB active_file:145280kB inactive_file:24869056kB unevictable:8528kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:2396kB dirty:634224kB writeback:0kB mapped:11496kB shmem:236kB slab_reclaimable:1976536kB slab_unreclaimable:600760kB kernel_stack:2408kB pagetables:7816kB unstable:376kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [104655.967852] lowmem_reserve[]: 0 0 0 0 [104655.967856] Node 1 Normal free:74664kB min:45092kB low:56364kB high:67636kB active_anon:407100kB inactive_anon:86908kB active_file:56340kB inactive_file:29356844kB unevictable:2064kB isolated(anon):0kB isolated(file):128kB present:33095680kB mlocked:2064kB dirty:92072kB writeback:1336kB mapped:11896kB shmem:904kB slab_reclaimable:1274452kB slab_unreclaimable:1127520kB kernel_stack:3104kB pagetables:6824kB unstable:50040kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [104655.967874] lowmem_reserve[]: 0 0 0 0 [104655.967878] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB [104655.967891] Node 0 DMA32: 30557*4kB 44*8kB 2*16kB 2*32kB 0*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 123956kB [104655.967904] Node 0 Normal: 16418*4kB 225*8kB 7*16kB 6*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 67968kB [104655.967917] Node 1 Normal: 13939*4kB 5*8kB 62*16kB 555*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 74548kB [104655.967930] 14082164 total pagecache pages [104655.967933] 3 pages in swap cache [104655.967935] Swap cache stats: add 76, delete 73, find 32/36 [104655.967938] Free swap = 8388532kB [104655.967940] Total swap = 8388604kB [104656.398724] 16777200 pages RAM [104656.398728] 393967 pages reserved [104656.398731] 7148738 pages shared [104656.398733] 9168632 pages non-shared [104656.398800] LustreError: 9219:0:(osc_request.c:2198:osc_build_rpc()) prep_req failed: -12 [104656.399153] LustreError: 9219:0:(osc_cache.c:2211:osc_check_rpcs()) Write request failed with -12 [357548.959693] LustreError: 102293:0:(ldlm_lock.c:223:ldlm_lock_put()) ASSERTION( (((( lock))->l_flags & (1ULL << 50)) != 0) ) failed: [357548.959716] LustreError: 102293:0:(ldlm_lock.c:223:ldlm_lock_put()) LBUG [357548.959724] Pid: 102293, comm: lustre_write.pl [357548.959726] [357548.959727] Call Trace: [357548.959755] [] dump_trace+0x75/0x300 [357548.959795] [] libcfs_debug_dumpstack+0x4a/0x70 [libcfs] [357548.959847] [] lbug_with_loc+0x3e/0xb0 [libcfs] [357548.959913] [] ldlm_lock_put+0x538/0x680 [ptlrpc] [357548.959966] [] ldlm_cli_cancel_list+0xef/0x3e0 [ptlrpc] [357548.960026] [] ldlm_prep_elc_req+0x2ad/0x490 [ptlrpc] [357548.960057] [] mdc_setattr+0x112/0x9b0 [mdc] [357548.960084] [] lmv_setattr+0x306/0x5f0 [lmv] [357548.960143] [] ll_md_setattr+0x1ff/0x9e0 [lustre] [357548.960230] [] ll_setattr_raw+0x252/0xb90 [lustre] [357548.960264] [] notify_change+0x19f/0x2f0 [357548.960283] [] do_truncate+0x57/0x80 [357548.960294] [] do_last+0x603/0x800 [357548.960302] [] path_openat+0xd9/0x420 [357548.960310] [] do_filp_open+0x4c/0xc0 [357548.960317] [] do_sys_open+0x171/0x1f0 [357548.960329] [] system_call_fastpath+0x16/0x1b [357548.960374] [<00007f2525444ff0>] 0x7f2525444ff0 [357548.960376] [357548.960705] Kernel panic - not syncing: LBUG [357548.960725] Pid: 102293, comm: lustre_write.pl Tainted: P NX 3.0.51-0.7.9-default #1 [357548.960743] Call Trace: [357548.960771] [] dump_trace+0x75/0x300 [357548.960792] [] dump_stack+0x69/0x6f [357548.960803] [] panic+0x93/0x201 [357548.960828] [] lbug_with_loc+0xa3/0xb0 [libcfs] [357548.960885] [] ldlm_lock_put+0x538/0x680 [ptlrpc] [357548.960939] [] ldlm_cli_cancel_list+0xef/0x3e0 [ptlrpc] [357548.961020] [] ldlm_prep_elc_req+0x2ad/0x490 [ptlrpc] [357548.961055] [] mdc_setattr+0x112/0x9b0 [mdc] [357548.961079] [] lmv_setattr+0x306/0x5f0 [lmv] [357548.961128] [] ll_md_setattr+0x1ff/0x9e0 [lustre] [357548.961179] [] ll_setattr_raw+0x252/0xb90 [lustre] [357548.961206] [] notify_change+0x19f/0x2f0 [357548.961214] [] do_truncate+0x57/0x80 [357548.961222] [] do_last+0x603/0x800 [357548.961233] [] path_openat+0xd9/0x420 [357548.961243] [] do_filp_open+0x4c/0xc0 [357548.961254] [] do_sys_open+0x171/0x1f0 [357548.961265] [] system_call_fastpath+0x16/0x1b [357548.961288] [<00007f2525444ff0>] 0x7f2525444fef " ================== " " TASK LIST " "================== " PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffffffff81a0b020 RU 0.0 0 0 [swapper] > 0 2 1 ffff88080f1b2300 RU 0.0 0 0 [kworker/0:0] > 0 2 2 ffff88080f1ce440 RU 0.0 0 0 [kworker/0:1] > 0 2 3 ffff88080f20a580 RU 0.0 0 0 [kworker/0:1] > 0 2 4 ffff88080f228040 RU 0.0 0 0 [kworker/0:1] > 0 2 5 ffff88080f262180 RU 0.0 0 0 [kworker/0:1] > 0 2 6 ffff88080f2802c0 RU 0.0 0 0 [kworker/0:1] > 0 2 7 ffff88080f2bc400 RU 0.0 0 0 [kworker/0:1] > 0 2 8 ffff88080f2c8480 RU 0.0 0 0 [kworker/0:1] 0 2 9 ffff88080f2d6500 RU 0.0 0 0 [kworker/0:1] > 0 2 10 ffff88080f2e2580 RU 0.0 0 0 [kworker/0:1] > 0 2 11 ffff88080f2ee600 RU 0.0 0 0 [kworker/0:1] > 0 2 12 ffff88080f2fc680 RU 0.0 0 0 [kworker/0:1] > 0 2 13 ffff88080f30a080 RU 0.0 0 0 [kworker/0:1] > 0 2 14 ffff88080f314100 RU 0.0 0 0 [kworker/0:1] > 0 2 15 ffff88080f322180 RU 0.0 0 0 [kworker/0:1] > 0 2 16 ffff88080f35e2c0 RU 0.0 0 0 [kworker/0:1] > 0 2 17 ffff88080f378400 RU 0.0 0 0 [kworker/0:1] > 0 2 18 ffff88080f3b6540 RU 0.0 0 0 [kworker/0:1] > 0 2 19 ffff88080f3d4680 RU 0.0 0 0 [kworker/0:1] > 0 2 20 ffff88080ec0e140 RU 0.0 0 0 [kworker/0:1] > 0 2 21 ffff88080ec2a280 RU 0.0 0 0 [kworker/0:1] > 0 2 22 ffff88080ec683c0 RU 0.0 0 0 [kworker/0:1] > 0 2 23 ffff88080ec84500 RU 0.0 0 0 [kworker/0:1] > 0 2 24 ffff88080ec90580 RU 0.0 0 0 [kworker/0:1] > 0 2 25 ffff88080ec9e600 RU 0.0 0 0 [kworker/0:1] > 0 2 26 ffff88080ecaa680 RU 0.0 0 0 [kworker/0:1] > 0 2 27 ffff88080ecb8080 RU 0.0 0 0 [kworker/0:1] > 0 2 28 ffff88080ecc4100 RU 0.0 0 0 [kworker/0:1] > 0 2 29 ffff88080ecee180 RU 0.0 0 0 [kworker/0:1] > 0 2 30 ffff88080ecfc200 RU 0.0 0 0 [kworker/0:1] > 0 2 31 ffff88080ed08280 RU 0.0 0 0 [kworker/0:1] 1 0 4 ffff88080f50a040 IN 0.0 10528 820 init 2 0 1 ffff88080f510080 IN 0.0 0 0 [kthreadd] 3 2 0 ffff88080f5360c0 IN 0.0 0 0 [ksoftirqd/0] 6 2 0 ffff88080f544180 IN 0.0 0 0 [migration/0] 7 2 0 ffff88080f1941c0 IN 0.0 0 0 [watchdog/0] 8 2 1 ffff88080f198200 IN 0.0 0 0 [migration/1] 10 2 1 ffff88080f1a8280 IN 0.0 0 0 [ksoftirqd/1] 12 2 1 ffff88080f1b6340 IN 0.0 0 0 [watchdog/1] 13 2 2 ffff88080f1ba380 IN 0.0 0 0 [migration/2] 15 2 2 ffff88080f1c6400 IN 0.0 0 0 [ksoftirqd/2] 16 2 2 ffff88080f1d4480 IN 0.0 0 0 [watchdog/2] 17 2 3 ffff88080f1d84c0 IN 0.0 0 0 [migration/3] 19 2 3 ffff88080f202540 IN 0.0 0 0 [ksoftirqd/3] 20 2 3 ffff88080f2105c0 IN 0.0 0 0 [watchdog/3] 21 2 4 ffff88080f216600 IN 0.0 0 0 [migration/4] 22 2 4 ffff88080f21a640 IN 0.0 0 0 [kworker/4:0] 23 2 4 ffff88080f21e680 IN 0.0 0 0 [ksoftirqd/4] 24 2 4 ffff88080f22c080 IN 0.0 0 0 [watchdog/4] 25 2 5 ffff88080f2300c0 IN 0.0 0 0 [migration/5] 26 2 5 ffff88080f238100 IN 0.0 0 0 [kworker/5:0] 27 2 5 ffff88080f25e140 IN 0.0 0 0 [ksoftirqd/5] 28 2 5 ffff88080f2681c0 IN 0.0 0 0 [watchdog/5] 29 2 6 ffff88080f26c200 IN 0.0 0 0 [migration/6] 31 2 6 ffff88080f276280 IN 0.0 0 0 [ksoftirqd/6] 32 2 6 ffff88080f284300 IN 0.0 0 0 [watchdog/6] 33 2 7 ffff88080f28a340 IN 0.0 0 0 [migration/7] 35 2 7 ffff88080f2b63c0 IN 0.0 0 0 [ksoftirqd/7] 36 2 7 ffff88080f2c0440 IN 0.0 0 0 [watchdog/7] 37 2 8 ffff88100f2d8040 IN 0.0 0 0 [migration/8] 39 2 8 ffff88100f3020c0 IN 0.0 0 0 [ksoftirqd/8] 40 2 8 ffff88080f2ce4c0 IN 0.0 0 0 [watchdog/8] 41 2 9 ffff88100f306100 IN 0.0 0 0 [migration/9] 42 2 9 ffff88100f30a140 IN 0.0 0 0 [kworker/9:0] 43 2 9 ffff88100f350180 IN 0.0 0 0 [ksoftirqd/9] 44 2 9 ffff88080f2da540 IN 0.0 0 0 [watchdog/9] 45 2 10 ffff88100f3541c0 IN 0.0 0 0 [migration/10] 46 2 10 ffff88100f358200 IN 0.0 0 0 [kworker/10:0] 47 2 10 ffff88100f39e240 IN 0.0 0 0 [ksoftirqd/10] 48 2 10 ffff88080f2e85c0 IN 0.0 0 0 [watchdog/10] 49 2 11 ffff88100f3a2280 IN 0.0 0 0 [migration/11] 50 2 11 ffff88100f3a62c0 IN 0.0 0 0 [kworker/11:0] 51 2 11 ffff88100f3ce300 IN 0.0 0 0 [ksoftirqd/11] 52 2 11 ffff88080f2f4640 IN 0.0 0 0 [watchdog/11] 53 2 12 ffff88100f3d2340 IN 0.0 0 0 [migration/12] 54 2 12 ffff88100f3d6380 IN 0.0 0 0 [kworker/12:0] 55 2 12 ffff88100f3fc3c0 IN 0.0 0 0 [ksoftirqd/12] 56 2 12 ffff88080f300040 IN 0.0 0 0 [watchdog/12] 57 2 13 ffff88100ec00400 IN 0.0 0 0 [migration/13] 59 2 13 ffff88100ec2c480 IN 0.0 0 0 [ksoftirqd/13] 60 2 13 ffff88080f30e0c0 IN 0.0 0 0 [watchdog/13] 61 2 14 ffff88100ec304c0 IN 0.0 0 0 [migration/14] 63 2 14 ffff88100ec3c540 IN 0.0 0 0 [ksoftirqd/14] 64 2 14 ffff88080f31a140 IN 0.0 0 0 [watchdog/14] 65 2 15 ffff88100ec40580 IN 0.0 0 0 [migration/15] 66 2 15 ffff88100ec445c0 IN 0.0 0 0 [kworker/15:0] 67 2 15 ffff88100ec6a600 IN 0.0 0 0 [ksoftirqd/15] 68 2 15 ffff88080f3261c0 IN 0.0 0 0 [watchdog/15] 69 2 16 ffff88080f32c200 IN 0.0 0 0 [migration/16] 70 2 16 ffff88080f330240 IN 0.0 0 0 [kworker/16:0] 71 2 16 ffff88080f358280 IN 0.0 0 0 [ksoftirqd/16] 72 2 16 ffff88080f362300 IN 0.0 0 0 [watchdog/16] 73 2 17 ffff88080f366340 IN 0.0 0 0 [migration/17] 75 2 17 ffff88080f3743c0 IN 0.0 0 0 [ksoftirqd/17] 76 2 17 ffff88080f380440 IN 0.0 0 0 [watchdog/17] 77 2 18 ffff88080f384480 IN 0.0 0 0 [migration/18] 79 2 18 ffff88080f3b2500 IN 0.0 0 0 [ksoftirqd/18] 80 2 18 ffff88080f3ba580 IN 0.0 0 0 [watchdog/18] 81 2 19 ffff88080f3c05c0 IN 0.0 0 0 [migration/19] 83 2 19 ffff88080f3ca640 IN 0.0 0 0 [ksoftirqd/19] 84 2 19 ffff88080f3d8040 IN 0.0 0 0 [watchdog/19] 85 2 20 ffff88080f3dc080 IN 0.0 0 0 [migration/20] 86 2 20 ffff88080f3e20c0 IN 0.0 0 0 [kworker/20:0] 87 2 20 ffff88080ec08100 IN 0.0 0 0 [ksoftirqd/20] 88 2 20 ffff88080ec14180 IN 0.0 0 0 [watchdog/20] 89 2 21 ffff88080ec181c0 IN 0.0 0 0 [migration/21] 90 2 21 ffff88080ec1c200 IN 0.0 0 0 [kworker/21:0] 91 2 21 ffff88080ec22240 IN 0.0 0 0 [ksoftirqd/21] 92 2 21 ffff88080ec2e2c0 IN 0.0 0 0 [watchdog/21] 93 2 22 ffff88080ec36300 IN 0.0 0 0 [migration/22] 94 2 22 ffff88080ec3a340 IN 0.0 0 0 [kworker/22:0] 95 2 22 ffff88080ec3e380 IN 0.0 0 0 [ksoftirqd/22] 96 2 22 ffff88080ec6c400 IN 0.0 0 0 [watchdog/22] 97 2 23 ffff88080ec70440 IN 0.0 0 0 [migration/23] 99 2 23 ffff88080ec804c0 IN 0.0 0 0 [ksoftirqd/23] 100 2 23 ffff88080ec8a540 IN 0.0 0 0 [watchdog/23] 101 2 24 ffff88100ec8e640 IN 0.0 0 0 [migration/24] 102 2 24 ffff88100ec92680 IN 0.0 0 0 [kworker/24:0] 103 2 24 ffff88100ec96040 IN 0.0 0 0 [ksoftirqd/24] 104 2 24 ffff88080ec965c0 IN 0.0 0 0 [watchdog/24] 105 2 25 ffff88100ec9e080 IN 0.0 0 0 [migration/25] 106 2 25 ffff88100eca20c0 IN 0.0 0 0 [kworker/25:0] 107 2 25 ffff88100ecc8100 IN 0.0 0 0 [ksoftirqd/25] 108 2 25 ffff88080eca2640 IN 0.0 0 0 [watchdog/25] 109 2 26 ffff88100eccc140 IN 0.0 0 0 [migration/26] 110 2 26 ffff88100ecd0180 IN 0.0 0 0 [kworker/26:0] 111 2 26 ffff88100ecd81c0 IN 0.0 0 0 [ksoftirqd/26] 112 2 26 ffff88080ecb0040 IN 0.0 0 0 [watchdog/26] 113 2 27 ffff88100ecdc200 IN 0.0 0 0 [migration/27] 114 2 27 ffff88100ece0240 IN 0.0 0 0 [kworker/27:0] 115 2 27 ffff88100ed08280 IN 0.0 0 0 [ksoftirqd/27] 116 2 27 ffff88080ecbc0c0 IN 0.0 0 0 [watchdog/27] 117 2 28 ffff88100ed0c2c0 IN 0.0 0 0 [migration/28] 118 2 28 ffff88100ed10300 IN 0.0 0 0 [kworker/28:0] 119 2 28 ffff88100ed18340 IN 0.0 0 0 [ksoftirqd/28] 120 2 28 ffff88080ecc8140 IN 0.0 0 0 [watchdog/28] 121 2 29 ffff88100ed1c380 IN 0.0 0 0 [migration/29] 122 2 29 ffff88100ed203c0 IN 0.0 0 0 [kworker/29:0] 123 2 29 ffff88100ed46400 IN 0.0 0 0 [ksoftirqd/29] 124 2 29 ffff88080ecf41c0 IN 0.0 0 0 [watchdog/29] 125 2 30 ffff88100ed4a440 IN 0.0 0 0 [migration/30] 126 2 30 ffff88100ed4e480 IN 0.0 0 0 [kworker/30:0] 127 2 30 ffff88100ed524c0 IN 0.0 0 0 [ksoftirqd/30] 128 2 30 ffff88080ed00240 IN 0.0 0 0 [watchdog/30] 129 2 31 ffff88100ed5a500 IN 0.0 0 0 [migration/31] 130 2 31 ffff88100ed5e540 IN 0.0 0 0 [kworker/31:0] 131 2 31 ffff88100ed86580 IN 0.0 0 0 [ksoftirqd/31] 132 2 31 ffff88080ed0e2c0 IN 0.0 0 0 [watchdog/31] 133 2 25 ffff88080ed58300 IN 0.0 0 0 [cpuset] 134 2 25 ffff88080ed80340 IN 0.0 0 0 [khelper] 135 2 25 ffff88080edd4380 IN 0.0 0 0 [netns] 136 2 17 ffff88080ee343c0 IN 0.0 0 0 [sync_supers] 137 2 8 ffff88080ee3a400 IN 0.0 0 0 [bdi-default] 138 2 25 ffff88080ee3e440 IN 0.0 0 0 [kintegrityd] 139 2 18 ffff88080eeb2480 IN 0.0 0 0 [kblockd] 140 2 25 ffff88080eb264c0 IN 0.0 0 0 [md] 141 2 25 ffff88080d05a500 IN 0.0 0 0 [khungtaskd] 142 2 10 ffff88100d79c5c0 IN 0.0 0 0 [kworker/10:1] 143 2 1 ffff88080d05e540 IN 0.0 0 0 [kswapd0] 144 2 24 ffff88080d0a4580 IN 0.0 0 0 [kswapd1] 145 2 25 ffff88080d0a85c0 IN 0.0 0 0 [ksmd] 146 2 10 ffff88080d0fe600 IN 0.0 0 0 [khugepaged] 147 2 10 ffff88080d114640 IN 0.0 0 0 [fsnotify_mark] 148 2 25 ffff88080d1c6680 IN 0.0 0 0 [crypto] 150 2 25 ffff88080d21e080 IN 0.0 0 0 [kthrotld] 151 2 25 ffff88100d0da600 IN 0.0 0 0 [kworker/25:1] 153 2 25 ffff88080d1ca040 IN 0.0 0 0 [kpsmoused] 154 2 11 ffff88100d312680 IN 0.0 0 0 [kworker/11:1] 161 2 9 ffff88100d3880c0 IN 0.0 0 0 [kworker/9:1] 178 2 17 ffff88080cd603c0 IN 0.0 0 0 [kworker/17:1] 199 2 26 ffff88100cd0a5c0 IN 0.0 0 0 [kworker/26:1] 203 2 25 ffff88080cf10680 IN 0.0 0 0 [scsi_eh_0] 204 2 6 ffff88080cc740c0 IN 0.0 0 0 [scsi_eh_1] 205 2 28 ffff88100cc36380 IN 0.0 0 0 [kworker/28:1] 206 2 27 ffff88100cc32340 IN 0.0 0 0 [kworker/27:1] 207 2 29 ffff88100d34c300 IN 0.0 0 0 [kworker/29:1] 209 2 31 ffff88100d358240 IN 0.0 0 0 [kworker/31:1] 210 2 25 ffff88080cdbe540 IN 0.0 0 0 [scsi_eh_2] 218 2 2 ffff88080ce06640 IN 0.0 0 0 [kworker/2:1] 222 2 25 ffff88080ccce200 IN 0.0 0 0 [ata_sff] 230 2 25 ffff88080cd102c0 IN 0.0 0 0 [scsi_eh_3] 231 2 24 ffff88080cca0140 IN 0.0 0 0 [scsi_eh_4] 232 2 9 ffff88080cca4180 IN 0.0 0 0 [scsi_eh_5] 233 2 8 ffff88080ccc81c0 IN 0.0 0 0 [scsi_eh_6] 234 2 10 ffff88080c5ca040 IN 0.0 0 0 [scsi_eh_7] 235 2 8 ffff88080c5dc080 IN 0.0 0 0 [scsi_eh_8] 268 2 16 ffff88080c7183c0 IN 0.0 0 0 [kworker/16:1] 269 2 19 ffff88080cd36380 IN 0.0 0 0 [kworker/19:1] 271 2 20 ffff88080c792400 IN 0.0 0 0 [kworker/20:1] 273 2 21 ffff88080c7dc480 IN 0.0 0 0 [kworker/21:1] 275 2 22 ffff88080c022500 IN 0.0 0 0 [kworker/22:1] 281 2 15 ffff88100cc5e400 IN 0.0 0 0 [kworker/15:1] 303 2 25 ffff88080cd32340 IN 0.0 0 0 [xfs_mru_cache] 304 2 8 ffff88080cd225c0 IN 0.0 0 0 [xfslogd] 305 2 25 ffff88080cc4c600 IN 0.0 0 0 [xfsdatad] 306 2 25 ffff88080c50a640 IN 0.0 0 0 [xfsconvertd] 350 2 25 ffff88080c5c0500 IN 0.0 0 0 [kmpath_rdacd] 694 2 0 ffff88100c07a540 IN 0.0 0 0 [khubd] 696 2 26 ffff88100c0a25c0 IN 0.0 0 0 [scsi_eh_9] 698 2 25 ffff88100c9ce4c0 IN 0.0 0 0 [scsi_wq_9] 913 2 15 ffff88100c586480 IN 0.0 0 0 [md126_raid1] 971 2 3 ffff88100c09e580 IN 0.0 0 0 [xfsbufd/md126p2] 972 2 10 ffff88100c484280 IN 0.0 0 0 [xfsaild/md126p2] 1045 1 16 ffff88080ccbe240 IN 0.0 10544 892 udevd 1110 2 17 ffff88100c742240 IN 0.0 0 0 [mlx4] 1774 2 17 ffff88100bf50240 IN 0.0 0 0 [cpe_migrate] 1796 2 17 ffff88100b13e580 IN 0.0 0 0 [*tsv_thread] 1831 2 18 ffff88080a8c0480 IN 0.0 0 0 [*xvm_gdaemon] 1834 2 25 ffff88080b8d6600 IN 0.0 0 0 [*xvm_fo_daemon] 1837 2 25 ffff88080c6ea380 IN 0.0 0 0 [*xvm_process_er] 1840 2 25 ffff88080a8ca500 IN 0.0 0 0 [*xvm_mr_revive_] 1843 2 26 ffff880809d00040 IN 0.0 0 0 [*xvm_mr_daemon] 2325 2 8 ffff88100c6061c0 IN 0.0 0 0 [xfsbufd/xvm-0] 2369 2 8 ffff88100b12e540 IN 0.0 0 0 [xfsaild/xvm-0] 2386 2 20 ffff88100bc28440 IN 0.0 0 0 [xfsbufd/xvm-1] 2411 2 3 ffff88100c130680 IN 0.0 0 0 [xfsbufd/xvm-2] 2415 2 12 ffff88100c7da3c0 IN 0.0 0 0 [xfsaild/xvm-2] 2416 2 16 ffff881009ef2400 IN 0.0 0 0 [xfsbufd/xvm-3] 2419 2 8 ffff881009e6c240 IN 0.0 0 0 [xfsaild/xvm-3] 2421 2 27 ffff88100b38e340 IN 0.0 0 0 [xfsaild/xvm-1] 2466 2 8 ffff88100c02e4c0 IN 0.0 0 0 [xfsbufd/xvm-4] 2478 2 1 ffff881009cee380 IN 0.0 0 0 [xfsbufd/xvm-6] 2479 2 10 ffff88100bd82600 IN 0.0 0 0 [xfsaild/xvm-6] 2480 2 9 ffff88100ccc8580 IN 0.0 0 0 [xfsaild/xvm-4] 2494 1 17 ffff88100c456140 IN 0.0 14692 10596 mdmon 2498 1 8 ffff88080c3de480 IN 0.0 14692 10596 mdmon 2549 2 22 ffff8810059ea1c0 IN 0.0 0 0 [kjournald] 2553 2 18 ffff881009c56300 IN 0.0 0 0 [rpciod] 2683 2 7 ffff88100c688180 IN 0.0 0 0 [flush-9:126] 3077 1 3 ffff8810059ee680 IN 0.0 3992 612 acpid 3092 1 9 ffff88080b450340 IN 0.0 49836 1380 dbus-daemon 3110 1 6 ffff881009dc4300 UN 0.0 15648 1648 syslog-ng 3113 1 7 ffff88080bbfa040 IN 0.0 4260 640 klogd 3663 1 8 ffff8807fee7e340 IN 0.0 67912 4796 hald 3666 1 1 ffff88100513a380 IN 0.0 108980 2268 console-kit-dae 3667 3663 0 ffff88080cce2340 IN 0.0 18152 1216 hald-runner 3668 1 9 ffff88080b2fc600 IN 0.0 108980 2268 console-kit-dae 3669 1 17 ffff88080a72a200 IN 0.0 108980 2268 console-kit-dae 3670 1 26 ffff88080b96c0c0 IN 0.0 108980 2268 console-kit-dae 3671 1 0 ffff880809a58100 IN 0.0 108980 2268 console-kit-dae 3672 1 1 ffff880809eac500 IN 0.0 108980 2268 console-kit-dae 3673 1 17 ffff88080a454680 IN 0.0 108980 2268 console-kit-dae 3675 1 17 ffff8807ff038640 IN 0.0 108980 2268 console-kit-dae 3676 1 17 ffff880809c8a580 IN 0.0 108980 2268 console-kit-dae 3677 1 17 ffff8807ff1a6200 IN 0.0 108980 2268 console-kit-dae 3678 1 0 ffff88080a6104c0 IN 0.0 108980 2268 console-kit-dae 3679 1 17 ffff88080c4d6100 IN 0.0 108980 2268 console-kit-dae 3680 1 1 ffff88080b9e4680 IN 0.0 108980 2268 console-kit-dae 3681 1 0 ffff88080c440200 IN 0.0 108980 2268 console-kit-dae 3682 1 0 ffff88080ccf8680 IN 0.0 108980 2268 console-kit-dae 3683 1 17 ffff88080cd6c440 IN 0.0 108980 2268 console-kit-dae 3684 1 17 ffff88080aa24240 IN 0.0 108980 2268 console-kit-dae 3685 1 0 ffff88080a7261c0 IN 0.0 108980 2268 console-kit-dae 3686 1 26 ffff88080a4c0100 IN 0.0 108980 2268 console-kit-dae 3687 1 17 ffff88080c3ce3c0 IN 0.0 108980 2268 console-kit-dae 3688 1 17 ffff88080a6fe180 IN 0.0 108980 2268 console-kit-dae 3689 1 0 ffff88080a562300 IN 0.0 108980 2268 console-kit-dae 3690 1 0 ffff8807fef66540 IN 0.0 108980 2268 console-kit-dae 3691 1 17 ffff88080a9d4140 IN 0.0 108980 2268 console-kit-dae 3692 1 0 ffff88080cd9a500 IN 0.0 108980 2268 console-kit-dae 3693 1 17 ffff88080a988040 IN 0.0 108980 2268 console-kit-dae 3694 1 0 ffff88080b2e63c0 IN 0.0 108980 2268 console-kit-dae 3695 1 17 ffff88080a9ca0c0 IN 0.0 108980 2268 console-kit-dae 3696 1 16 ffff88080aaaa400 IN 0.0 108980 2268 console-kit-dae 3697 1 0 ffff8807ffed6500 IN 0.0 108980 2268 console-kit-dae 3698 1 17 ffff8807fef2c380 IN 0.0 108980 2268 console-kit-dae 3699 1 1 ffff8807fef624c0 IN 0.0 108980 2268 console-kit-dae 3700 1 17 ffff880809eb2540 IN 0.0 108980 2268 console-kit-dae 3701 1 0 ffff880809cb4600 IN 0.0 108980 2268 console-kit-dae 3702 1 0 ffff88080a4ca180 IN 0.0 108980 2268 console-kit-dae 3703 1 1 ffff8807fed76140 IN 0.0 108980 2268 console-kit-dae 3704 1 17 ffff88080aa3c340 IN 0.0 108980 2268 console-kit-dae 3705 1 0 ffff88080a9165c0 IN 0.0 108980 2268 console-kit-dae 3706 1 17 ffff88080a4a0040 IN 0.0 108980 2268 console-kit-dae 3707 1 1 ffff88080b18a200 IN 0.0 108980 2268 console-kit-dae 3708 1 0 ffff88080a6a8040 IN 0.0 108980 2268 console-kit-dae 3709 1 0 ffff88080b3ec200 IN 0.0 108980 2268 console-kit-dae 3710 1 17 ffff8807ff24e600 IN 0.0 108980 2268 console-kit-dae 3711 1 1 ffff88080b44e100 IN 0.0 108980 2268 console-kit-dae 3712 1 0 ffff88080b3c2180 IN 0.0 108980 2268 console-kit-dae 3713 1 17 ffff880809c86540 IN 0.0 108980 2268 console-kit-dae 3714 1 0 ffff880809d2a100 IN 0.0 108980 2268 console-kit-dae 3715 1 17 ffff88080a4ee1c0 IN 0.0 108980 2268 console-kit-dae 3716 1 0 ffff88080aa302c0 IN 0.0 108980 2268 console-kit-dae 3717 1 17 ffff88080cdf6380 IN 0.0 108980 2268 console-kit-dae 3718 1 17 ffff88080c478400 IN 0.0 108980 2268 console-kit-dae 3719 1 0 ffff88080a6d20c0 IN 0.0 108980 2268 console-kit-dae 3720 1 17 ffff88080a5e0400 IN 0.0 108980 2268 console-kit-dae 3721 1 0 ffff88080a6725c0 IN 0.0 108980 2268 console-kit-dae 3722 1 17 ffff88080b5a6480 IN 0.0 108980 2268 console-kit-dae 3723 1 0 ffff880809c7a180 IN 0.0 108980 2268 console-kit-dae 3724 1 17 ffff88080bb4a480 IN 0.0 108980 2268 console-kit-dae 3725 1 0 ffff88080b594400 IN 0.0 108980 2268 console-kit-dae 3726 1 17 ffff8807fecfc240 IN 0.0 108980 2268 console-kit-dae 3727 1 0 ffff8807ff120280 IN 0.0 108980 2268 console-kit-dae 3728 1 0 ffff8807ff1242c0 IN 0.0 108980 2268 console-kit-dae 3729 1 17 ffff8807ff290300 IN 0.0 108980 2268 console-kit-dae 3762 3667 8 ffff88080b31a040 IN 0.0 34760 1072 hald-addon-inpu 3810 3667 12 ffff88100c3e6640 IN 0.0 30200 920 hald-addon-acpi 3846 2 17 ffff88100d3b6180 IN 0.0 0 0 [mthca_catas] 3849 2 17 ffff8810052d2500 IN 0.0 0 0 [mlx4_ib] 3850 2 2 ffff881005636140 IN 0.0 0 0 [ib_mad1] 3851 2 18 ffff88100bd40100 IN 0.0 0 0 [ib_mad2] 3855 2 17 ffff8810052e6600 IN 0.0 0 0 [mlx4_en] 3859 2 17 ffff881009e34040 IN 0.0 0 0 [iw_cxgb3] 3865 2 17 ffff881004c54640 IN 0.0 0 0 [ib_mcast] 3866 2 26 ffff88100b1dc2c0 IN 0.0 0 0 [ib_cm] 3867 2 17 ffff8810051c4180 IN 0.0 0 0 [ipoib] 3989 1 7 ffff88080a566340 IN 0.0 59632 2016 nmbd 3990 3989 16 ffff8810051181c0 IN 0.0 67460 2244 nmbd 4146 2 17 ffff88100c73c200 IN 0.0 0 0 [ib_addr] 4147 2 17 ffff88100cc74100 IN 0.0 0 0 [iw_cm_wq] 4148 2 17 ffff88100b26e040 IN 0.0 0 0 [rdma_cm] 5696 1 25 ffff8807fb630340 IN 0.0 25592 504 eventmond 5698 5696 11 ffff88100c1fc0c0 IN 0.0 301324 3128 eventmond 5699 5696 4 ffff88100d0f8440 IN 0.0 301324 3128 eventmond 5728 1 21 ffff88080aaa0380 IN 0.0 8352 4524 haveged 5742 1 2 ffff881005828240 IN 0.0 70708 4884 snmpd 5743 1 1 ffff8807ff3642c0 IN 0.0 70708 4884 snmpd 5748 1 2 ffff8807ff5465c0 IN 0.0 121304 7564 lsi_mrdsnmpagen 5775 5748 23 ffff88080a534400 IN 0.0 21540 1332 lsi_mrdsnmpagen 5776 1 7 ffff8810052224c0 IN 0.0 121304 7564 lsi_mrdsnmpagen 5777 1 24 ffff88080bb5c0c0 IN 0.0 121304 7564 lsi_mrdsnmpagen 5801 1 11 ffff88080c6c63c0 IN 0.0 16596 604 memlogd 5840 1 5 ffff88100cc80440 IN 0.0 18892 960 rpcbind 5862 1 8 ffff88100c4fe2c0 IN 0.0 11288 1308 sh 5866 5862 0 ffff88100563a100 IN 0.6 16817004 378080 java 5887 5862 0 ffff8807fb6de100 IN 0.6 16817004 378080 java 5894 5862 20 ffff8810053b0200 IN 0.6 16817004 378080 java 5895 5862 30 ffff881005882380 IN 0.6 16817004 378080 java 5896 5862 25 ffff8810052e25c0 IN 0.6 16817004 378080 java 5897 5862 3 ffff881005a282c0 IN 0.6 16817004 378080 java 5898 5862 17 ffff881004cbc280 IN 0.6 16817004 378080 java 5899 5862 18 ffff881005b18240 IN 0.6 16817004 378080 java 5900 5862 28 ffff88100593e5c0 IN 0.6 16817004 378080 java 5901 5862 11 ffff8810057ea580 IN 0.6 16817004 378080 java 5902 5862 24 ffff88100d9fc240 IN 0.6 16817004 378080 java 5903 5862 31 ffff88100c1b02c0 IN 0.6 16817004 378080 java 5904 5862 2 ffff88100d3b2140 IN 0.6 16817004 378080 java 5905 5862 4 ffff8810053c6240 IN 0.6 16817004 378080 java 5906 5862 10 ffff881004dee180 IN 0.6 16817004 378080 java 5907 5862 0 ffff8810051104c0 IN 0.6 16817004 378080 java 5908 5862 7 ffff881004d64280 IN 0.6 16817004 378080 java 5909 5862 9 ffff881009c9e440 IN 0.6 16817004 378080 java 5910 5862 8 ffff881009e24040 IN 0.6 16817004 378080 java 5911 5862 13 ffff88100495a480 IN 0.6 16817004 378080 java 5912 5862 11 ffff881005734300 IN 0.6 16817004 378080 java 5913 5862 14 ffff8810059ec200 IN 0.6 16817004 378080 java 5914 5862 6 ffff88100ccd0080 IN 0.6 16817004 378080 java 5915 5862 16 ffff881005588400 IN 0.6 16817004 378080 java 5916 5862 21 ffff8810052ce4c0 IN 0.6 16817004 378080 java 5930 5862 15 ffff88100c5a8100 IN 0.6 16817004 378080 java 5941 5862 8 ffff88100de841c0 IN 0.6 16817004 378080 java 5943 5862 3 ffff88100c45a180 IN 0.6 16817004 378080 java 5971 5862 17 ffff881009e20180 IN 0.6 16817004 378080 java 5972 5862 4 ffff881005a04280 IN 0.6 16817004 378080 java 5973 5862 7 ffff88100b3963c0 IN 0.6 16817004 378080 java 5974 5862 18 ffff88100db2e480 IN 0.6 16817004 378080 java 5975 5862 6 ffff88100dafc440 IN 0.6 16817004 378080 java 6016 5862 22 ffff8807fb652140 IN 0.6 16817004 378080 java 6018 5862 8 ffff8807ff842380 IN 0.6 16817004 378080 java 6031 6018 6 ffff88100c858600 IN 0.1 16762192 63512 java 6032 5862 6 ffff88080aad2480 IN 0.6 16817004 378080 java 6052 5862 27 ffff8807ff3624c0 IN 0.6 16817004 378080 java 6053 5862 17 ffff88080a60c480 IN 0.6 16817004 378080 java 6054 5862 27 ffff8807fb63c0c0 IN 0.6 16817004 378080 java 6055 5862 17 ffff880809c1c440 IN 0.6 16817004 378080 java 6056 5862 27 ffff8807ff36e2c0 IN 0.6 16817004 378080 java 6057 5862 17 ffff88080aad6200 IN 0.6 16817004 378080 java 6058 5862 27 ffff88080a4324c0 IN 0.6 16817004 378080 java 6059 5862 17 ffff88080b0321c0 IN 0.6 16817004 378080 java 6060 5862 27 ffff88080a8c64c0 IN 0.6 16817004 378080 java 6061 5862 17 ffff88080b4ee240 IN 0.6 16817004 378080 java 6066 1 18 ffff88100c36c540 IN 0.0 16596 604 memlogd 6067 1 22 ffff88080a67a640 IN 0.0 33028 10460 espdbd 6069 1 22 ffff88100bc485c0 IN 0.0 33028 10460 espdbd 6070 1 16 ffff88100528a480 IN 0.0 33028 10460 espdbd 6089 5696 8 ffff88080c474280 IN 0.0 301324 3128 eventmond 6091 5696 21 ffff881005726080 IN 0.0 301324 3128 eventmond 6092 5696 6 ffff88080aaae440 IN 0.0 301324 3128 eventmond 6093 5696 3 ffff88080a4c4140 IN 0.0 301324 3128 eventmond 6095 1 7 ffff88100d316040 IN 0.0 33028 10460 espdbd 6136 2 18 ffff88100bf4c200 IN 0.0 0 0 [nfsiod] 6144 1 0 ffff8807ff252180 IN 0.0 55876 1772 rpc.idmapd 6221 5696 17 ffff88100c594500 IN 0.0 301324 3128 eventmond 6222 5696 25 ffff8807fbbf2680 IN 0.0 301324 3128 eventmond 6956 1 8 ffff88080cda4240 IN 0.0 17208 1024 pmcd 7042 1 25 ffff880809d04080 IN 0.0 10560 716 pmproxy 7078 1 0 ffff8807f7244140 IN 0.0 15208 800 sgi_irqbalance 7111 1 0 ffff881004e7c540 IN 0.0 65012 1380 sshd 7134 1 28 ffff88100c1f8380 IN 0.0 21672 1420 ntpd 7148 1 23 ffff8807ff51a300 IN 0.0 67564 2328 cupsd 7163 1 8 ffff8807fb700200 IN 0.0 277300 3352 nscd 7164 1 16 ffff881004f58600 IN 0.0 277300 3352 nscd 7165 1 16 ffff88100c002440 IN 0.0 277300 3352 nscd 7166 1 16 ffff881009dac300 IN 0.0 277300 3352 nscd 7167 1 22 ffff88100c6e8580 IN 0.0 277300 3352 nscd 7168 1 0 ffff88100c052500 IN 0.0 277300 3352 nscd 7169 1 1 ffff88100c60c0c0 IN 0.0 277300 3352 nscd 7170 1 16 ffff881005418040 IN 0.0 277300 3352 nscd 7171 1 1 ffff88100d6cc600 IN 0.0 277300 3352 nscd 7496 1 9 ffff88080b96a640 IN 0.0 11404 996 pmie 7538 1 8 ffff88080dfa82c0 IN 0.0 16688 804 automount 7539 1 17 ffff881009d82040 IN 0.0 16688 804 automount 7540 1 26 ffff8810051ce240 IN 0.0 16688 804 automount 7571 1 17 ffff88080dfaa240 IN 0.0 21760 1476 rpc.mountd 7575 1 8 ffff88080a868380 IN 0.0 17424 1252 rpc.statd 7583 2 17 ffff88100d2de600 IN 0.0 0 0 [lockd] 7584 2 26 ffff881005640140 IN 0.0 0 0 [nfsd4] 7585 2 3 ffff8810050e2100 IN 0.0 0 0 [nfsd4_callbacks] 7586 2 9 ffff88100d4001c0 IN 0.0 0 0 [nfsd] 7587 2 6 ffff88100c7fe400 IN 0.0 0 0 [nfsd] 7588 2 3 ffff88100c0a6600 IN 0.0 0 0 [nfsd] 7589 2 2 ffff88100b1a4080 IN 0.0 0 0 [nfsd] 7590 2 18 ffff881009c160c0 IN 0.0 0 0 [nfsd] 7591 2 15 ffff88100b0da2c0 IN 0.0 0 0 [nfsd] 7592 2 18 ffff88100518a480 IN 0.0 0 0 [nfsd] 7593 2 6 ffff881009cc80c0 IN 0.0 0 0 [nfsd] 7594 2 9 ffff8810058de4c0 IN 0.0 0 0 [nfsd] 7595 2 7 ffff88100bf06180 IN 0.0 0 0 [nfsd] 7596 2 8 ffff88100c6465c0 IN 0.0 0 0 [nfsd] 7597 2 18 ffff88100523e600 IN 0.0 0 0 [nfsd] 7598 2 6 ffff88100bf4e500 IN 0.0 0 0 [nfsd] 7599 2 7 ffff881009e88100 IN 0.0 0 0 [nfsd] 7600 2 8 ffff88100d6c05c0 IN 0.0 0 0 [nfsd] 7601 2 0 ffff8810056ca280 IN 0.0 0 0 [nfsd] 7602 2 6 ffff88100dba24c0 IN 0.0 0 0 [nfsd] 7603 2 3 ffff881005b14200 IN 0.0 0 0 [nfsd] 7604 2 7 ffff8810051d65c0 IN 0.0 0 0 [nfsd] 7605 2 6 ffff8810053ac1c0 IN 0.0 0 0 [nfsd] 7606 2 8 ffff881005738600 IN 0.0 0 0 [nfsd] 7607 2 6 ffff88100c2c8600 IN 0.0 0 0 [nfsd] 7608 2 24 ffff8810052ea480 IN 0.0 0 0 [nfsd] 7609 2 6 ffff88100b174040 IN 0.0 0 0 [nfsd] 7610 2 8 ffff88100ccd81c0 IN 0.0 0 0 [nfsd] 7611 2 9 ffff881005958640 IN 0.0 0 0 [nfsd] 7612 2 3 ffff881004d763c0 IN 0.0 0 0 [nfsd] 7613 2 18 ffff88100c546100 IN 0.0 0 0 [nfsd] 7614 2 6 ffff88100d6f4200 IN 0.0 0 0 [nfsd] 7615 2 8 ffff88100585a280 IN 0.0 0 0 [nfsd] 7616 2 7 ffff88100c5fe380 IN 0.0 0 0 [nfsd] 7617 2 18 ffff881009c883c0 IN 0.0 0 0 [nfsd] 7618 2 3 ffff881004d14440 IN 0.0 0 0 [nfsd] 7619 2 18 ffff8810053d6340 IN 0.0 0 0 [nfsd] 7620 2 8 ffff88100d570280 IN 0.0 0 0 [nfsd] 7621 2 18 ffff88100c56c400 IN 0.0 0 0 [nfsd] 7622 2 7 ffff881009c002c0 IN 0.0 0 0 [nfsd] 7623 2 18 ffff88100b3ba580 IN 0.0 0 0 [nfsd] 7624 2 8 ffff88100daaa200 IN 0.0 0 0 [nfsd] 7625 2 3 ffff8810056a8300 IN 0.0 0 0 [nfsd] 7626 2 7 ffff881009ef4340 IN 0.0 0 0 [nfsd] 7627 2 18 ffff8810054ee380 IN 0.0 0 0 [nfsd] 7628 2 18 ffff88100d4ea3c0 IN 0.0 0 0 [nfsd] 7629 2 8 ffff88100565c400 IN 0.0 0 0 [nfsd] 7630 2 8 ffff881005990440 IN 0.0 0 0 [nfsd] 7631 2 6 ffff881005500480 IN 0.0 0 0 [nfsd] 7632 2 18 ffff8810048ba4c0 IN 0.0 0 0 [nfsd] 7633 2 2 ffff88100516a500 IN 0.0 0 0 [nfsd] 7634 2 3 ffff88100dfc6540 IN 0.0 0 0 [nfsd] 7635 2 6 ffff88100b0f6580 IN 0.0 0 0 [nfsd] 7636 2 18 ffff8810055d65c0 IN 0.0 0 0 [nfsd] 7637 2 18 ffff88100c240600 IN 0.0 0 0 [nfsd] 7638 2 6 ffff881009c10640 IN 0.0 0 0 [nfsd] 7639 2 7 ffff881009d5a680 IN 0.0 0 0 [nfsd] 7640 2 9 ffff881005520040 IN 0.0 0 0 [nfsd] 7641 2 9 ffff8810055dc080 IN 0.0 0 0 [nfsd] 7642 2 18 ffff88100b2520c0 IN 0.0 0 0 [nfsd] 7643 2 18 ffff88100d992100 IN 0.0 0 0 [nfsd] 7644 2 7 ffff88100d51e140 IN 0.0 0 0 [nfsd] 7645 2 18 ffff881004806180 IN 0.0 0 0 [nfsd] 7646 2 6 ffff881004c561c0 IN 0.0 0 0 [nfsd] 7647 2 25 ffff8810051e6200 IN 0.0 0 0 [nfsd] 7648 2 7 ffff88100b074240 IN 0.0 0 0 [nfsd] 7649 2 2 ffff881004dd0280 IN 0.0 0 0 [nfsd] 7710 1 5 ffff88100c10a640 IN 0.0 38208 1980 master 7739 7710 21 ffff8807ffeaa340 IN 0.0 40208 2032 qmgr 7741 1 7 ffff88100dd86500 IN 0.0 81292 2300 smbd 7755 7741 16 ffff8807fece6080 IN 0.0 81292 1604 smbd 7794 1 17 ffff8807fb646040 IN 0.0 66840 2508 gdm 7795 7794 24 ffff88080c4305c0 IN 0.0 60308 9152 gdm-simple-slav 7796 7795 1 ffff881004dbc580 IN 0.0 136032 23204 X 7798 1 26 ffff8807fb66a300 IN 0.0 108980 2268 console-kit-dae 7816 1 14 ffff88080c428580 IN 0.0 14788 696 cron 7943 1 7 ffff88100c368240 IN 0.0 16600 1008 smartd 7995 1 22 ffff881009f20600 IN 0.0 4484 764 mingetty 7996 1 8 ffff8810049ae340 IN 0.0 4484 760 mingetty 7997 1 3 ffff88100bff0440 IN 0.0 4484 760 mingetty 7998 1 7 ffff88100bdbe240 IN 0.0 4484 764 mingetty 7999 1 9 ffff88100cace480 IN 0.0 4484 764 mingetty 8000 1 16 ffff881005620040 IN 0.0 4484 760 mingetty 8128 1 0 ffff88100c770540 IN 0.0 108980 2268 console-kit-dae 8139 1 8 ffff881004c80640 IN 0.0 22192 524 dbus-launch 8140 1 0 ffff8810048dc540 IN 0.0 13448 764 dbus-daemon 8141 7795 8 ffff88100c922580 IN 0.0 158208 4348 gnome-session 8152 1 12 ffff88100bf84340 IN 0.0 39460 4296 gconfd-2 8153 8141 1 ffff8807fb6fc640 IN 0.0 127436 3564 at-spi-registry 8154 7795 0 ffff88100b08c240 IN 0.0 158208 4348 gnome-session 8157 1 16 ffff881005154580 IN 0.0 272344 7204 gnome-settings- 8159 1 8 ffff881004f3a340 IN 0.0 129984 4208 bonobo-activati 8160 1 7 ffff88080cd72480 IN 0.0 129984 4208 bonobo-activati 8162 1 9 ffff88100c528340 IN 0.0 272344 7204 gnome-settings- 8163 1 9 ffff8807ff2984c0 IN 0.0 230132 6152 gnome-power-man 8167 1 22 ffff88080c1262c0 IN 0.0 43416 1604 gvfsd 8168 8141 0 ffff88080c1160c0 IN 0.0 216876 6052 metacity 8170 8141 25 ffff8807d3bc0180 IN 0.0 251476 14012 gdm-simple-gree 8175 1 7 ffff88100c38a100 IN 0.0 186544 1920 pulseaudio 8178 1 26 ffff88080c108180 IN 0.0 186544 1920 null-sink 8191 2 17 ffff88100cca4480 IN 0.0 0 0 [kauditd] 8322 79422 7 ffff8800990a81c0 IN 0.0 70712 2488 su 8323 8322 1 ffff88009909e180 IN 0.0 13472 2288 bash 8771 7111 18 ffff88080a6b2580 IN 0.0 94816 2976 sshd 8773 8771 0 ffff88080b040600 IN 0.0 13804 2676 bash 8829 1045 11 ffff8807d34c2540 IN 0.0 10540 868 udevd 9139 2 26 ffff880ff4730500 IN 0.0 0 0 [cfs_rh_00] 9140 2 26 ffff8810055aa300 IN 0.0 0 0 [cfs_rh_01] 9141 2 17 ffff88100bc56680 IN 0.0 0 0 [cfs_rh_02] 9142 2 17 ffff880ff4042280 IN 0.0 0 0 [cfs_rh_03] 9177 2 25 ffff88100d882300 IN 0.0 0 0 [obd_zombid] 9178 2 17 ffff880ff402a100 IN 0.0 0 0 [ptlrpc_hr00_000] 9179 2 17 ffff88100ca16640 IN 0.0 0 0 [ptlrpc_hr00_001] 9180 2 17 ffff880ff43c03c0 IN 0.0 0 0 [ptlrpc_hr00_002] 9181 2 17 ffff880ff404a300 IN 0.0 0 0 [ptlrpc_hr00_003] 9182 2 4 ffff88100d8cc440 IN 0.0 0 0 [ptlrpc_hr01_000] 9183 2 4 ffff880ff4322600 IN 0.0 0 0 [ptlrpc_hr01_001] 9184 2 4 ffff880ff436c140 IN 0.0 0 0 [ptlrpc_hr01_002] 9185 2 4 ffff880ff43b22c0 IN 0.0 0 0 [ptlrpc_hr01_003] 9186 2 8 ffff880ff4228180 IN 0.0 0 0 [ptlrpc_hr02_000] 9187 2 8 ffff880ff4198300 IN 0.0 0 0 [ptlrpc_hr02_001] 9188 2 8 ffff880ff4182540 IN 0.0 0 0 [ptlrpc_hr02_002] 9189 2 8 ffff88100da363c0 IN 0.0 0 0 [ptlrpc_hr02_003] 9190 2 12 ffff880ff479c540 IN 0.0 0 0 [ptlrpc_hr03_000] 9191 2 12 ffff880ff431e5c0 IN 0.0 0 0 [ptlrpc_hr03_001] 9192 2 12 ffff880ff4266440 IN 0.0 0 0 [ptlrpc_hr03_002] 9193 2 12 ffff880ff475c140 IN 0.0 0 0 [ptlrpc_hr03_003] 9196 2 11 ffff880ff419c340 IN 0.0 0 0 [kiblnd_connd] 9197 2 18 ffff88100d42c4c0 IN 0.0 0 0 [kiblnd_sd_00_00] 9198 2 2 ffff880ff4760180 IN 0.0 0 0 [kiblnd_sd_00_01] 9199 2 1 ffff880ff4032180 IN 0.0 0 0 [kiblnd_sd_00_02] 9200 2 3 ffff880ff4778300 IN 0.0 0 0 [kiblnd_sd_00_03] 9201 2 4 ffff880ff4350040 IN 0.0 0 0 [kiblnd_sd_01_00] 9202 2 4 ffff880ff4788400 IN 0.0 0 0 [kiblnd_sd_01_01] 9203 2 4 ffff880ff476c240 IN 0.0 0 0 [kiblnd_sd_01_02] 9204 2 4 ffff881005136140 IN 0.0 0 0 [kiblnd_sd_01_03] 9205 2 25 ffff880ff43bc380 IN 0.0 0 0 [kiblnd_sd_02_00] 9206 2 25 ffff880ff43d8480 IN 0.0 0 0 [kiblnd_sd_02_01] 9207 2 25 ffff880ff47540c0 IN 0.0 0 0 [kiblnd_sd_02_02] 9208 2 25 ffff88100b1a81c0 IN 0.0 0 0 [kiblnd_sd_02_03] 9209 2 14 ffff88100494e200 IN 0.0 0 0 [kiblnd_sd_03_00] 9210 2 28 ffff88100c27e240 IN 0.0 0 0 [kiblnd_sd_03_01] 9211 2 13 ffff8810058260c0 IN 0.0 0 0 [kiblnd_sd_03_02] 9212 2 31 ffff88100d6b2040 IN 0.0 0 0 [kiblnd_sd_03_03] 9213 2 22 ffff88100c2b0080 IN 0.0 0 0 [router_checker] 9214 2 9 ffff88100daec100 IN 0.0 0 0 [ptlrpcd_rcv] 9215 2 28 ffff88100b3c0180 IN 0.0 0 0 [ptlrpcd_0] 9216 2 22 ffff880ff47ac640 IN 0.0 0 0 [ptlrpcd_1] 9217 2 18 ffff880ff472c4c0 IN 0.0 0 0 [ptlrpcd_2] 9218 2 0 ffff880ff4354080 IN 0.0 0 0 [ptlrpcd_3] 9219 2 0 ffff880ff4798500 IN 0.0 0 0 [ptlrpcd_4] 9220 2 6 ffff880ff4770280 IN 0.0 0 0 [ptlrpcd_5] 9221 2 10 ffff880ff404c1c0 IN 0.0 0 0 [ptlrpcd_6] 9222 2 16 ffff880ff4740600 IN 0.0 0 0 [ptlrpcd_7] 9223 2 16 ffff880ff47641c0 IN 0.0 0 0 [ptlrpcd_8] 9224 2 10 ffff880ff4750080 IN 0.0 0 0 [ptlrpcd_9] 9225 2 10 ffff880ff443a580 IN 0.0 0 0 [ptlrpcd_10] 9226 2 29 ffff880ff443e5c0 IN 0.0 0 0 [ptlrpcd_11] 9227 2 1 ffff880ff4442600 IN 0.0 0 0 [ptlrpcd_12] 9228 2 12 ffff880ff4446640 IN 0.0 0 0 [ptlrpcd_13] 9229 2 11 ffff880ff444a680 IN 0.0 0 0 [ptlrpcd_14] 9230 2 27 ffff880ff444e040 IN 0.0 0 0 [ptlrpcd_15] 9231 2 8 ffff880ff4452080 IN 0.0 0 0 [ptlrpcd_16] 9232 2 18 ffff880ff44560c0 IN 0.0 0 0 [ptlrpcd_17] 9233 2 18 ffff880ff445a100 IN 0.0 0 0 [ptlrpcd_18] 9234 2 2 ffff880ff445e140 IN 0.0 0 0 [ptlrpcd_19] 9235 2 28 ffff880ff4462180 IN 0.0 0 0 [ptlrpcd_20] 9236 2 19 ffff880ff44661c0 IN 0.0 0 0 [ptlrpcd_21] 9237 2 25 ffff880ff446a200 IN 0.0 0 0 [ptlrpcd_22] 9238 2 1 ffff880ff446e240 IN 0.0 0 0 [ptlrpcd_23] 9239 2 19 ffff880ff4472280 IN 0.0 0 0 [ptlrpcd_24] 9240 2 12 ffff880ff44762c0 IN 0.0 0 0 [ptlrpcd_25] 9241 2 27 ffff880ff447a300 IN 0.0 0 0 [ptlrpcd_26] 9242 2 29 ffff880ff447e340 IN 0.0 0 0 [ptlrpcd_27] 9243 2 23 ffff880ff4482380 IN 0.0 0 0 [ptlrpcd_28] 9244 2 13 ffff880ff44863c0 IN 0.0 0 0 [ptlrpcd_29] 9245 2 26 ffff880ff448a400 IN 0.0 0 0 [ptlrpcd_30] 9246 2 28 ffff880ff448e440 IN 0.0 0 0 [ptlrpcd_31] 9247 2 24 ffff880ff4492480 IN 0.0 0 0 [ll_ping] 9248 2 0 ffff880ff44964c0 IN 0.0 0 0 [sptlrpc_gc] 9249 2 25 ffff881009db85c0 IN 0.0 0 0 [ll_capa] 9252 2 2 ffff880ff404e340 IN 0.0 0 0 [ldlm_cb00_000] 9253 2 9 ffff880ff40361c0 IN 0.0 0 0 [lc_watchdogd] 9254 2 16 ffff880ff47a45c0 IN 0.0 0 0 [ldlm_cb00_001] 9255 2 4 ffff880ff4346640 IN 0.0 0 0 [ldlm_cb01_000] 9256 2 4 ffff88100c1a2540 IN 0.0 0 0 [ldlm_cb01_001] 9257 2 8 ffff88100bd8c2c0 IN 0.0 0 0 [ldlm_cb02_000] 9258 2 8 ffff880ff438e280 IN 0.0 0 0 [ldlm_cb02_001] 9259 2 12 ffff880ff483c500 IN 0.0 0 0 [ldlm_cb03_000] 9260 2 13 ffff880ff5ac8540 IN 0.0 0 0 [ldlm_cb03_001] 9261 2 17 ffff880ff5acc580 IN 0.0 0 0 [ldlm_bl_00] 9262 2 8 ffff880ff44f05c0 IN 0.0 0 0 [ldlm_bl_01] 9263 2 10 ffff880ff44f4600 IN 0.0 0 0 [ll_cfg_requeue] 9267 2 17 ffff880ff5b9c680 IN 0.0 0 0 [ll_close] 9419 1045 0 ffff880ff4a043c0 IN 0.0 10540 868 udevd 9843 7111 0 ffff88005dbfc2c0 IN 0.0 94816 2900 sshd 9846 9843 22 ffff8807fc2fe380 IN 0.0 94948 1756 sshd 9847 9846 3 ffff8801823a03c0 IN 0.0 14272 2788 tcsh 12000 7111 7 ffff88007429e240 IN 0.0 97128 3784 sshd 12002 12000 20 ffff880727a50340 IN 0.0 97128 2212 sshd 12003 12002 24 ffff880610826380 IN 0.0 13448 2844 bash 12034 12003 17 ffff880b254e4200 IN 0.0 53376 2428 screen 14762 2 12 ffff8804c5f52240 IN 0.0 0 0 [ldlm_cb03_006] 20263 2 2 ffff8803fdd8e680 IN 0.0 0 0 [kworker/2:0] 25971 2 23 ffff88005287e200 IN 0.0 0 0 [kworker/23:2] 26174 2 1 ffff880158dc4440 IN 0.0 0 0 [kworker/1:2] 26194 2 14 ffff880b3552a540 IN 0.0 0 0 [kworker/14:1] 26212 2 23 ffff880001bf0600 IN 0.0 0 0 [kworker/23:0] 26220 2 12 ffff88000f4b2140 IN 0.0 0 0 [kworker/12:2] 26230 2 30 ffff880328768300 IN 0.0 0 0 [kworker/30:2] 26280 2 4 ffff880419aae200 IN 0.0 0 0 [kworker/4:1] 26283 2 14 ffff8800ba9922c0 IN 0.0 0 0 [kworker/14:2] 26289 2 13 ffff88100cc1e280 IN 0.0 0 0 [kworker/13:1] 26356 2 5 ffff8800105d0440 IN 0.0 0 0 [kworker/5:2] 26360 2 13 ffff88004523e500 IN 0.0 0 0 [kworker/13:0] 33801 2 7 ffff8804a1b34300 IN 0.0 0 0 [kworker/7:1] 38365 2 17 ffff880f84e7e540 IN 0.0 0 0 [ldlm_bl_02] 38366 2 29 ffff880f84e68580 IN 0.0 0 0 [ldlm_cb03_002] 38367 2 27 ffff880f700845c0 IN 0.0 0 0 [ldlm_bl_02] 44620 2 17 ffff880f22b32100 IN 0.0 0 0 [ldlm_cb00_002] 65658 2 0 ffff880e6de083c0 IN 0.0 0 0 [ldlm_cb00_003] 65659 2 0 ffff880f04a12680 IN 0.0 0 0 [ldlm_bl_04] 65660 2 15 ffff880cca120180 IN 0.0 0 0 [ldlm_bl_04] 65661 2 8 ffff88087dfc0540 IN 0.0 0 0 [ldlm_bl_04] 65662 2 13 ffff880eece8a400 IN 0.0 0 0 [ldlm_cb03_003] 65663 2 28 ffff880f2f978380 IN 0.0 0 0 [ldlm_bl_04] 65664 2 18 ffff88089640a600 IN 0.0 0 0 [ldlm_bl_05] 65665 2 8 ffff880ae34bc340 IN 0.0 0 0 [ldlm_bl_06] 65666 2 22 ffff880e3e492240 IN 0.0 0 0 [ldlm_bl_08] 65667 2 29 ffff880c3001c440 IN 0.0 0 0 [ldlm_bl_08] 65668 2 5 ffff88086f90e2c0 IN 0.0 0 0 [ldlm_bl_08] 65669 2 11 ffff880e6e0b0040 IN 0.0 0 0 [ldlm_bl_08] 65670 2 26 ffff880846bce340 IN 0.0 0 0 [ldlm_bl_09] 65671 2 6 ffff880fe77942c0 IN 0.0 0 0 [ldlm_bl_09] 65672 2 1 ffff880e748c0140 IN 0.0 0 0 [ldlm_bl_11] 65673 2 1 ffff88084f1c0680 IN 0.0 0 0 [ldlm_bl_11] 65674 2 17 ffff880cd07bc540 IN 0.0 0 0 [ldlm_bl_11] 65675 2 8 ffff88086c1c4580 IN 0.0 0 0 [ldlm_bl_12] 65676 2 1 ffff880cac85a340 IN 0.0 0 0 [ldlm_bl_12] 65679 2 16 ffff880f0f0ba200 IN 0.0 0 0 [ldlm_cb00_004] 65682 2 1 ffff880e698c03c0 IN 0.0 0 0 [ldlm_cb00_005] 65785 2 28 ffff880e8d4a01c0 IN 0.0 0 0 [ldlm_cb03_004] 77532 2 18 ffff880889606280 IN 0.0 0 0 [ldlm_cb00_006] 77553 2 12 ffff8807aca10580 IN 0.0 0 0 [ldlm_cb03_005] 77897 2 2 ffff880879b0e440 IN 0.0 0 0 [ldlm_cb00_007] 79421 1 14 ffff88010bd8c440 IN 0.1 110964 58820 screen 79422 79421 8 ffff8804d0ee4500 IN 0.0 12920 2048 bash 79430 79421 15 ffff880713650240 IN 0.0 12920 2048 bash 79437 79430 16 ffff88086d35e3c0 IN 0.0 70712 2480 su 79438 79437 9 ffff88055f8e82c0 IN 0.0 13576 2340 bash 79658 79421 8 ffff8800562a01c0 IN 0.0 12920 2056 bash 79666 79658 0 ffff880cb9c50680 IN 0.0 70712 2480 su 79667 79666 10 ffff88004fcf4280 IN 0.0 13576 2316 bash 87205 2 7 ffff880eb2444680 IN 0.0 0 0 [kworker/7:2] 88464 1 8 ffff8800699b8080 IN 0.0 18024 3168 bash 88749 88464 1 ffff880968b56540 IN 0.0 5044 552 tail 88760 2 3 ffff88014ee54380 IN 0.0 0 0 [kworker/3:2] 89380 2 3 ffff88005c928540 IN 0.0 0 0 [kworker/3:0] 93575 2 1 ffff88039f44c400 IN 0.0 0 0 [kworker/1:1] 93613 2 17 ffff880014552340 IN 0.0 0 0 [kworker/17:2] 93682 2 24 ffff880e1beee040 IN 0.0 0 0 [kworker/24:1] 96326 2 19 ffff880f0ff0a1c0 IN 0.0 0 0 [kworker/19:0] 99729 2 18 ffff88087bf78500 IN 0.0 0 0 [kworker/18:1] 99804 2 18 ffff880117f76180 IN 0.0 0 0 [kworker/18:2] 100268 2 8 ffff880a0dc902c0 IN 0.0 0 0 [kworker/8:1] 100275 2 6 ffff880ca5c1e3c0 IN 0.0 0 0 [kworker/6:1] 100628 2 6 ffff8800a4f5a3c0 IN 0.0 0 0 [kworker/6:0] 101063 2 0 ffff8800622964c0 IN 0.0 0 0 [kworker/0:0] 101650 7710 7 ffff88000a5f2040 IN 0.0 40056 2196 pickup 101716 2 0 ffff880065f46640 IN 0.0 0 0 [kworker/0:2] 101810 2 7 ffff88035fbba400 IN 0.0 0 0 [kworker/u:1] 101835 2 8 ffff8808969d65c0 IN 0.0 0 0 [kworker/8:2] 101851 2 9 ffff88028582e340 IN 0.0 0 0 [flush-lustre-1] 102075 2 18 ffff88054b21e380 IN 0.0 0 0 [kworker/u:2] 102235 2 2 ffff8804f1910040 IN 0.0 0 0 [kworker/u:0] 102241 7795 24 ffff880001988440 IN 0.0 110600 4544 gdm-session-wor 102265 7816 18 ffff8808ab816580 IN 0.0 67396 2124 cron 102267 102265 27 ffff880302f3a400 IN 0.0 11288 1320 sh 102270 102267 9 ffff8808ab80a600 IN 0.0 32776 7868 lustre_monitor. > 102293 102270 9 ffff8808ce46e080 RU 0.0 19960 3448 lustre_write.pl 114839 2 13 ffff88076599e480 IN 0.0 0 0 [ldlm_cb03_007] " ================== " " STACK TRACE " "================== " PID: 0 TASK: ffffffff81a0b020 CPU: 0 COMMAND: "swapper" #0 [ffff88083f607eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f607ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f607ef0] notify_die at ffffffff8144899d #3 [ffff88083f607f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f607f40] do_nmi at ffffffff81446058 #5 [ffff88083f607f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffffffff81a01ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837c0395 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffff81a00010 RDI: ffffffff81a01fd8 RBP: 0000000000000030 R8: 0000000000000b4d R9: 0000000000000001 R10: 00000000000017cd R11: 0000000000000000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffa00200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffffffff81a01ea8] intel_idle at ffffffff812a8f61 #7 [ffffffff81a01ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffffffff81a01f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f1b2300 CPU: 1 COMMAND: "kworker/0:0" #0 [ffff88083f627eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f627ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f627ef0] notify_die at ffffffff8144899d #3 [ffff88083f627f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f627f40] do_nmi at ffffffff81446058 #5 [ffff88083f627f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f1b5ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837afbe1 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f1b4010 RDI: ffff88080f1b5fd8 RBP: 0000000000000030 R8: 0000000000036fb9 R9: 0000000000000001 R10: 0000000000001785 R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffa20200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f1b5ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f1b5ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f1b5f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f1ce440 CPU: 2 COMMAND: "kworker/0:1" #0 [ffff88083f647eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f647ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f647ef0] notify_die at ffffffff8144899d #3 [ffff88083f647f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f647f40] do_nmi at ffffffff81446058 #5 [ffff88083f647f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f1d1ea8 RFLAGS: 00000046 RAX: 0000000000000010 RBX: 1348f5ab83815dc8 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f1d0010 RDI: ffff88080f1d1fd8 RBP: 0000000000000010 R8: 00000000000003aa R9: 000000000000000b R10: 0000000000001990 R11: 0000000000000000 R12: 0000000000000004 R13: 0000000000000002 R14: ffffe8f7ffa40140 R15: 0000000000000002 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f1d1ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f1d1ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f1d1f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f20a580 CPU: 3 COMMAND: "kworker/0:1" #0 [ffff88083f667eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f667ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f667ef0] notify_die at ffffffff8144899d #3 [ffff88083f667f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f667f40] do_nmi at ffffffff81446058 #5 [ffff88083f667f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f20dea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837b8088 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f20c010 RDI: ffff88080f20dfd8 RBP: 0000000000000030 R8: 0000000000000bde R9: 0000000000000001 R10: 0000000000001497 R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffa60200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f20dea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f20def0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f20df30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f228040 CPU: 4 COMMAND: "kworker/0:1" #0 [ffff88083f687eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f687ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f687ef0] notify_die at ffffffff8144899d #3 [ffff88083f687f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f687f40] do_nmi at ffffffff81446058 #5 [ffff88083f687f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f22bea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab7c6f4113 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f22a010 RDI: ffff88080f22bfd8 RBP: 0000000000000030 R8: 0000000000238504 R9: 0000000000000015 R10: 00000000000018e9 R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffa80200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f22bea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f22bef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f22bf30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f262180 CPU: 5 COMMAND: "kworker/0:1" #0 [ffff88083f6a7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f6a7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f6a7ef0] notify_die at ffffffff8144899d #3 [ffff88083f6a7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f6a7f40] do_nmi at ffffffff81446058 #5 [ffff88083f6a7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f265ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab376cad66 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f264010 RDI: ffff88080f265fd8 RBP: 0000000000000030 R8: 00000000003cfb8f R9: 0000000000000001 R10: 000000000000181a R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffaa0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f265ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f265ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f265f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f2802c0 CPU: 6 COMMAND: "kworker/0:1" #0 [ffff88083f6c7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f6c7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f6c7ef0] notify_die at ffffffff8144899d #3 [ffff88083f6c7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f6c7f40] do_nmi at ffffffff81446058 #5 [ffff88083f6c7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f283ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab83951e73 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f282010 RDI: ffff88080f283fd8 RBP: 0000000000000030 R8: 0000000000007161 R9: 0000000000000001 R10: 0000000000000568 R11: 0000000000000001 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffac0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f283ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f283ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f283f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f2bc400 CPU: 7 COMMAND: "kworker/0:1" #0 [ffff88083f6e7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f6e7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f6e7ef0] notify_die at ffffffff8144899d #3 [ffff88083f6e7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f6e7f40] do_nmi at ffffffff81446058 #5 [ffff88083f6e7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f2bfea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab83992982 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f2be010 RDI: ffff88080f2bffd8 RBP: 0000000000000030 R8: 0000000000000dc2 R9: 0000000000000001 R10: 000000000000009d R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffae0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f2bfea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f2bfef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f2bff30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f2c8480 CPU: 8 COMMAND: "kworker/0:1" #0 [ffff88103f207eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f207ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f207ef0] notify_die at ffffffff8144899d #3 [ffff88103f207f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f207f40] do_nmi at ffffffff81446058 #5 [ffff88103f207f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f2cbea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab839a57bc RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f2ca010 RDI: ffff88080f2cbfd8 RBP: 0000000000000030 R8: 0000000000000551 R9: 0000000000000001 R10: 000000000000150d R11: 0000000000000000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff600200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f2cbea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f2cbef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f2cbf30] cpu_idle at ffffffff810020e6 PID: 102293 TASK: ffff8808ce46e080 CPU: 9 COMMAND: "lustre_write.pl" #0 [ffff880864a7d7e0] machine_kexec at ffffffff8102676e #1 [ffff880864a7d830] crash_kexec at ffffffff810a3a3a #2 [ffff880864a7d900] panic at ffffffff81441e53 #3 [ffff880864a7d980] lbug_with_loc at ffffffffa0961dc3 [libcfs] #4 [ffff880864a7d9a0] ldlm_lock_put at ffffffffa0bf9648 [ptlrpc] #5 [ffff880864a7d9c0] ldlm_cli_cancel_list at ffffffffa0c0c83f [ptlrpc] #6 [ffff880864a7da00] ldlm_prep_elc_req at ffffffffa0c0e8cd [ptlrpc] #7 [ffff880864a7da70] mdc_setattr at ffffffffa0d4b6e2 [mdc] #8 [ffff880864a7db00] lmv_setattr at ffffffffa0f88ab6 [lmv] #9 [ffff880864a7db60] ll_md_setattr at ffffffffa0ece50f [lustre] #10 [ffff880864a7dc00] ll_setattr_raw at ffffffffa0ecef42 [lustre] #11 [ffff880864a7dca0] notify_change at ffffffff8116a88f #12 [ffff880864a7dd00] do_truncate at ffffffff8114f307 #13 [ffff880864a7dd70] do_last at ffffffff8115d8c3 #14 [ffff880864a7ddd0] path_openat at ffffffff8115e729 #15 [ffff880864a7de60] do_filp_open at ffffffff8115ebac #16 [ffff880864a7df30] do_sys_open at ffffffff8114f891 #17 [ffff880864a7df80] system_call_fastpath at ffffffff8144ca12 RIP: 00007f2525444ff0 RSP: 00007fff96b5a428 RFLAGS: 00010246 RAX: 0000000000000002 RBX: ffffffff8144ca12 RCX: 0000000000795078 RDX: 00000000000001b6 RSI: 0000000000000241 RDI: 00000000007b5d18 RBP: 000000000078e010 R8: 00007fff96b5a360 R9: 00000000ffffffff R10: 000000000050ab40 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000241 R14: 00000000ffffffff R15: 00007fff96b5a360 ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b PID: 0 TASK: ffff88080f2e2580 CPU: 10 COMMAND: "kworker/0:1" #0 [ffff88103f247eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f247ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f247ef0] notify_die at ffffffff8144899d #3 [ffff88103f247f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f247f40] do_nmi at ffffffff81446058 #5 [ffff88103f247f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f2e5ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837cff1e RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f2e4010 RDI: ffff88080f2e5fd8 RBP: 0000000000000030 R8: 0000000000000f8a R9: 0000000000000001 R10: 0000000000001af9 R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff640200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f2e5ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f2e5ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f2e5f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f2ee600 CPU: 11 COMMAND: "kworker/0:1" #0 [ffff88103f267eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f267ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f267ef0] notify_die at ffffffff8144899d #3 [ffff88103f267f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f267f40] do_nmi at ffffffff81446058 #5 [ffff88103f267f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f2f1ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab8384d026 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f2f0010 RDI: ffff88080f2f1fd8 RBP: 0000000000000030 R8: 0000000000000b8c R9: 0000000000000001 R10: 0000000000001ad0 R11: 00000000000024c6 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff660200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f2f1ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f2f1ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f2f1f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f2fc680 CPU: 12 COMMAND: "kworker/0:1" #0 [ffff88103f287eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f287ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f287ef0] notify_die at ffffffff8144899d #3 [ffff88103f287f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f287f40] do_nmi at ffffffff81446058 #5 [ffff88103f287f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f2ffea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837d88b8 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f2fe010 RDI: ffff88080f2fffd8 RBP: 0000000000000030 R8: 0000000000000f12 R9: 0000000000000001 R10: 0000000000001b25 R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff680200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f2ffea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f2ffef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f2fff30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f30a080 CPU: 13 COMMAND: "kworker/0:1" #0 [ffff88103f2a7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f2a7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f2a7ef0] notify_die at ffffffff8144899d #3 [ffff88103f2a7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f2a7f40] do_nmi at ffffffff81446058 #5 [ffff88103f2a7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f30dea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837d8fdc RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f30c010 RDI: ffff88080f30dfd8 RBP: 0000000000000030 R8: 0000000000000d63 R9: 0000000000000001 R10: 000000000000190e R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff6a0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f30dea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f30def0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f30df30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f314100 CPU: 14 COMMAND: "kworker/0:1" #0 [ffff88103f2c7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f2c7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f2c7ef0] notify_die at ffffffff8144899d #3 [ffff88103f2c7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f2c7f40] do_nmi at ffffffff81446058 #5 [ffff88103f2c7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f317ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837de119 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f316010 RDI: ffff88080f317fd8 RBP: 0000000000000030 R8: 0000000000000c49 R9: 0000000000000001 R10: 000000000000166d R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff6c0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f317ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f317ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f317f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f322180 CPU: 15 COMMAND: "kworker/0:1" #0 [ffff88103f2e7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f2e7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f2e7ef0] notify_die at ffffffff8144899d #3 [ffff88103f2e7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f2e7f40] do_nmi at ffffffff81446058 #5 [ffff88103f2e7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f325ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab83961fcf RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f324010 RDI: ffff88080f325fd8 RBP: 0000000000000030 R8: 000000000000073d R9: 0000000000000001 R10: 0000000000001856 R11: 0000000000000000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff6e0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f325ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f325ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f325f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f35e2c0 CPU: 16 COMMAND: "kworker/0:1" #0 [ffff88083f707eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f707ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f707ef0] notify_die at ffffffff8144899d #3 [ffff88083f707f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f707f40] do_nmi at ffffffff81446058 #5 [ffff88083f707f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f361ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837e8557 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f360010 RDI: ffff88080f361fd8 RBP: 0000000000000030 R8: 0000000000000f8e R9: 0000000000000001 R10: 0000000000001acb R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffb00200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f361ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f361ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f361f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f378400 CPU: 17 COMMAND: "kworker/0:1" #0 [ffff88083f727eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f727ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f727ef0] notify_die at ffffffff8144899d #3 [ffff88083f727f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f727f40] do_nmi at ffffffff81446058 #5 [ffff88083f727f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f37bea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837ec031 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f37a010 RDI: ffff88080f37bfd8 RBP: 0000000000000030 R8: 0000000000000ef1 R9: 0000000000000001 R10: 00000000000019bf R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffb20200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f37bea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f37bef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f37bf30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f3b6540 CPU: 18 COMMAND: "kworker/0:1" #0 [ffff88083f747eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f747ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f747ef0] notify_die at ffffffff8144899d #3 [ffff88083f747f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f747f40] do_nmi at ffffffff81446058 #5 [ffff88083f747f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f3b9ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab8376a34d RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f3b8010 RDI: ffff88080f3b9fd8 RBP: 0000000000000030 R8: 00000000000017f5 R9: 000000000000000b R10: 000000000000051e R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffb40200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f3b9ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f3b9ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f3b9f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080f3d4680 CPU: 19 COMMAND: "kworker/0:1" #0 [ffff88083f767eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f767ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f767ef0] notify_die at ffffffff8144899d #3 [ffff88083f767f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f767f40] do_nmi at ffffffff81446058 #5 [ffff88083f767f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080f3d7ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837f32aa RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080f3d6010 RDI: ffff88080f3d7fd8 RBP: 0000000000000030 R8: 0000000000000a73 R9: 0000000000000001 R10: 00000000000011a0 R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffb60200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080f3d7ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080f3d7ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080f3d7f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ec0e140 CPU: 20 COMMAND: "kworker/0:1" #0 [ffff88083f787eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f787ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f787ef0] notify_die at ffffffff8144899d #3 [ffff88083f787f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f787f40] do_nmi at ffffffff81446058 #5 [ffff88083f787f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ec11ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab49c9266b RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ec10010 RDI: ffff88080ec11fd8 RBP: 0000000000000030 R8: 000000000033149d R9: 0000000000000001 R10: 00000000000015bc R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffb80200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ec11ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ec11ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ec11f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ec2a280 CPU: 21 COMMAND: "kworker/0:1" #0 [ffff88083f7a7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f7a7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f7a7ef0] notify_die at ffffffff8144899d #3 [ffff88083f7a7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f7a7f40] do_nmi at ffffffff81446058 #5 [ffff88083f7a7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ec2dea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837a21b4 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ec2c010 RDI: ffff88080ec2dfd8 RBP: 0000000000000030 R8: 0000000000160747 R9: 0000000000000001 R10: 0000000000000f28 R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffba0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ec2dea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ec2def0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ec2df30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ec683c0 CPU: 22 COMMAND: "kworker/0:1" #0 [ffff88083f7c7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f7c7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f7c7ef0] notify_die at ffffffff8144899d #3 [ffff88083f7c7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f7c7f40] do_nmi at ffffffff81446058 #5 [ffff88083f7c7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ec6bea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab6628801a RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ec6a010 RDI: ffff88080ec6bfd8 RBP: 0000000000000030 R8: 00000000000d2625 R9: 0000000000000001 R10: 0000000000001794 R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffbc0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ec6bea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ec6bef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ec6bf30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ec84500 CPU: 23 COMMAND: "kworker/0:1" #0 [ffff88083f7e7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88083f7e7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88083f7e7ef0] notify_die at ffffffff8144899d #3 [ffff88083f7e7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88083f7e7f40] do_nmi at ffffffff81446058 #5 [ffff88083f7e7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ec87ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab662874f8 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ec86010 RDI: ffff88080ec87fd8 RBP: 0000000000000030 R8: 00000000000e1e8f R9: 0000000000000001 R10: 000000000000199d R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8f7ffbe0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ec87ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ec87ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ec87f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ec90580 CPU: 24 COMMAND: "kworker/0:1" #0 [ffff88103f307eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f307ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f307ef0] notify_die at ffffffff8144899d #3 [ffff88103f307f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f307f40] do_nmi at ffffffff81446058 #5 [ffff88103f307f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ec93ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab83805fb2 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ec92010 RDI: ffff88080ec93fd8 RBP: 0000000000000030 R8: 0000000000000f8a R9: 0000000000000001 R10: 0000000000001b10 R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff700200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ec93ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ec93ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ec93f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ec9e600 CPU: 25 COMMAND: "kworker/0:1" #0 [ffff88103f327eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f327ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f327ef0] notify_die at ffffffff8144899d #3 [ffff88103f327f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f327f40] do_nmi at ffffffff81446058 #5 [ffff88103f327f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080eca1ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab8382a7f5 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080eca0010 RDI: ffff88080eca1fd8 RBP: 0000000000000030 R8: 0000000000037279 R9: 000000000000000b R10: 0000000000000f03 R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff720200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080eca1ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080eca1ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080eca1f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ecaa680 CPU: 26 COMMAND: "kworker/0:1" #0 [ffff88103f347eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f347ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f347ef0] notify_die at ffffffff8144899d #3 [ffff88103f347f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f347f40] do_nmi at ffffffff81446058 #5 [ffff88103f347f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ecadea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab8383f01d RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ecac010 RDI: ffff88080ecadfd8 RBP: 0000000000000030 R8: 0000000000000dc4 R9: 0000000000000001 R10: 0000000000001580 R11: 00000000000024c6 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff740200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ecadea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ecadef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ecadf30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ecb8080 CPU: 27 COMMAND: "kworker/0:1" #0 [ffff88103f367eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f367ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f367ef0] notify_die at ffffffff8144899d #3 [ffff88103f367f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f367f40] do_nmi at ffffffff81446058 #5 [ffff88103f367f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ecbbea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab820cf07a RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ecba010 RDI: ffff88080ecbbfd8 RBP: 0000000000000030 R8: 000000000000cabf R9: 0000000000000001 R10: 0000000000001b27 R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff760200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ecbbea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ecbbef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ecbbf30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ecc4100 CPU: 28 COMMAND: "kworker/0:1" #0 [ffff88103f387eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f387ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f387ef0] notify_die at ffffffff8144899d #3 [ffff88103f387f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f387f40] do_nmi at ffffffff81446058 #5 [ffff88103f387f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ecc7ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab837b945c RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ecc6010 RDI: ffff88080ecc7fd8 RBP: 0000000000000030 R8: 000000000003a2db R9: 0000000000000001 R10: 00000000000010a4 R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff780200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ecc7ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ecc7ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ecc7f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ecee180 CPU: 29 COMMAND: "kworker/0:1" #0 [ffff88103f3a7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f3a7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f3a7ef0] notify_die at ffffffff8144899d #3 [ffff88103f3a7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f3a7f40] do_nmi at ffffffff81446058 #5 [ffff88103f3a7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ecf1ea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab7fb0d5d2 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ecf0010 RDI: ffff88080ecf1fd8 RBP: 0000000000000030 R8: 0000000000000f90 R9: 0000000000000001 R10: 0000000000001b20 R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff7a0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ecf1ea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ecf1ef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ecf1f30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ecfc200 CPU: 30 COMMAND: "kworker/0:1" #0 [ffff88103f3c7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f3c7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f3c7ef0] notify_die at ffffffff8144899d #3 [ffff88103f3c7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f3c7f40] do_nmi at ffffffff81446058 #5 [ffff88103f3c7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ecffea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab8199aedd RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ecfe010 RDI: ffff88080ecfffd8 RBP: 0000000000000030 R8: 0000000000000f8d R9: 0000000000000001 R10: 0000000000001aef R11: ffffffff813643a0 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff7c0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ecffea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ecffef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ecfff30] cpu_idle at ffffffff810020e6 PID: 0 TASK: ffff88080ed08280 CPU: 31 COMMAND: "kworker/0:1" #0 [ffff88103f3e7eb0] crash_nmi_callback at ffffffff8101eb8f #1 [ffff88103f3e7ec0] notifier_call_chain at ffffffff81448907 #2 [ffff88103f3e7ef0] notify_die at ffffffff8144899d #3 [ffff88103f3e7f20] default_do_nmi at ffffffff81445e93 #4 [ffff88103f3e7f40] do_nmi at ffffffff81446058 #5 [ffff88103f3e7f50] nmi at ffffffff814456a0 [exception RIP: intel_idle+241] RIP: ffffffff812a8f61 RSP: ffff88080ed0bea8 RFLAGS: 00000046 RAX: 0000000000000030 RBX: 1348f5ab8378e559 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff88080ed0a010 RDI: ffff88080ed0bfd8 RBP: 0000000000000030 R8: 0000000000171433 R9: 0000000000000001 R10: 0000000000000f1b R11: ffffffff81020000 R12: 0000000000000010 R13: 0000000000000004 R14: ffffe8ffff7e0200 R15: 0000000000000004 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #6 [ffff88080ed0bea8] intel_idle at ffffffff812a8f61 #7 [ffff88080ed0bef0] cpuidle_idle_call at ffffffff8136348b #8 [ffff88080ed0bf30] cpu_idle at ffffffff810020e6 " ================== " " TASK PANIC'ed " "================== " PID: 102293 TASK: ffff8808ce46e080 CPU: 9 COMMAND: "lustre_write.pl" #0 [ffff880864a7d7e0] machine_kexec at ffffffff8102676e #1 [ffff880864a7d830] crash_kexec at ffffffff810a3a3a #2 [ffff880864a7d900] panic at ffffffff81441e53 #3 [ffff880864a7d980] lbug_with_loc at ffffffffa0961dc3 [libcfs] #4 [ffff880864a7d9a0] ldlm_lock_put at ffffffffa0bf9648 [ptlrpc] #5 [ffff880864a7d9c0] ldlm_cli_cancel_list at ffffffffa0c0c83f [ptlrpc] #6 [ffff880864a7da00] ldlm_prep_elc_req at ffffffffa0c0e8cd [ptlrpc] #7 [ffff880864a7da70] mdc_setattr at ffffffffa0d4b6e2 [mdc] #8 [ffff880864a7db00] lmv_setattr at ffffffffa0f88ab6 [lmv] #9 [ffff880864a7db60] ll_md_setattr at ffffffffa0ece50f [lustre] #10 [ffff880864a7dc00] ll_setattr_raw at ffffffffa0ecef42 [lustre] #11 [ffff880864a7dca0] notify_change at ffffffff8116a88f #12 [ffff880864a7dd00] do_truncate at ffffffff8114f307 #13 [ffff880864a7dd70] do_last at ffffffff8115d8c3 #14 [ffff880864a7ddd0] path_openat at ffffffff8115e729 #15 [ffff880864a7de60] do_filp_open at ffffffff8115ebac #16 [ffff880864a7df30] do_sys_open at ffffffff8114f891 #17 [ffff880864a7df80] system_call_fastpath at ffffffff8144ca12 RIP: 00007f2525444ff0 RSP: 00007fff96b5a428 RFLAGS: 00010246 RAX: 0000000000000002 RBX: ffffffff8144ca12 RCX: 0000000000795078 RDX: 00000000000001b6 RSI: 0000000000000241 RDI: 00000000007b5d18 RBP: 000000000078e010 R8: 00007fff96b5a360 R9: 00000000ffffffff R10: 000000000050ab40 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000241 R14: 00000000ffffffff R15: 00007fff96b5a360 ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b