2013-02-21 13:08:35 rockstar invoked oom-killer: gfp_mask=0x0, order=0, oom_adj=0, oom_score_adj=0 2013-02-21 13:08:35 rockstar cpuset=/ mems_allowed=0-1 2013-02-21 13:08:35 Pid: 16439, comm: rockstar Tainted: P --------------- 2.6.32-279.9.1.el6.x86_64 #1 2013-02-21 13:08:36 Call Trace: 2013-02-21 13:08:36 [] ? cpuset_print_task_mems_allowed+0x91/0xb0 2013-02-21 13:08:36 [] ? dump_header+0x90/0x1b0 2013-02-21 13:08:36 [] ? __do_fault+0x1/0x510 2013-02-21 13:08:36 [] ? check_panic_on_oom+0x4e/0x80 2013-02-21 13:08:36 [] ? out_of_memory+0x1bb/0x3c0 2013-02-21 13:08:36 [] ? pagefault_out_of_memory+0xc5/0x110 2013-02-21 13:08:36 [] ? mm_fault_error+0xb2/0x1a0 2013-02-21 13:08:36 [] ? __do_page_fault+0x33b/0x480 2013-02-21 13:08:36 [] ? __switch_to+0x26e/0x320 2013-02-21 13:08:36 [] ? thread_return+0x4e/0x76e 2013-02-21 13:08:36 [] ? utrace_resume+0x8b/0x110 2013-02-21 13:08:36 [] ? reschedule_interrupt+0xe/0x20 2013-02-21 13:08:36 [] ? do_page_fault+0x3e/0xa0 2013-02-21 13:08:36 [] ? page_fault+0x25/0x30 2013-02-21 13:08:36 Mem-Info: 2013-02-21 13:08:36 Node 0 DMA per-cpu: 2013-02-21 13:08:36 CPU 0: hi: 0, btch: 1 usd: 0 2013-02-21 13:08:36 CPU 1: hi: 0, btch: 1 usd: 0 2013-02-21 13:08:36 CPU 2: hi: 0, btch: 1 usd: 0 2013-02-21 13:08:36 CPU 3: hi: 0, btch: 1 usd: 0 2013-02-21 13:08:37 CPU 4: hi: 0, btch: 1 usd: 0 2013-02-21 13:08:37 CPU 5: hi: 0, btch: 1 usd: 0 2013-02-21 13:08:37 CPU 6: hi: 0, btch: 1 usd: 0 2013-02-21 13:08:37 CPU 7: hi: 0, btch: 1 usd: 0 2013-02-21 13:08:37 Node 0 DMA32 per-cpu: 2013-02-21 13:08:37 CPU 0: hi: 186, btch: 31 usd: 177 2013-02-21 13:08:37 CPU 1: hi: 186, btch: 31 usd: 156 2013-02-21 13:08:37 CPU 2: hi: 186, btch: 31 usd: 2 2013-02-21 13:08:37 CPU 3: hi: 186, btch: 31 usd: 0 2013-02-21 13:08:37 CPU 4: hi: 186, btch: 31 usd: 0 2013-02-21 13:08:37 CPU 5: hi: 186, btch: 31 usd: 0 2013-02-21 13:08:37 CPU 6: hi: 186, btch: 31 usd: 0 2013-02-21 13:08:37 CPU 7: hi: 186, btch: 31 usd: 0 2013-02-21 13:08:37 Node 0 Normal per-cpu: 2013-02-21 13:08:37 CPU 0: hi: 186, btch: 31 usd: 5 2013-02-21 13:08:37 CPU 1: hi: 186, btch: 31 usd: 45 2013-02-21 13:08:37 CPU 2: hi: 186, btch: 31 usd: 45 2013-02-21 13:08:37 CPU 3: hi: 186, btch: 31 usd: 45 2013-02-21 13:08:37 CPU 4: hi: 186, btch: 31 usd: 138 2013-02-21 13:08:37 CPU 5: hi: 186, btch: 31 usd: 165 2013-02-21 13:08:37 CPU 6: hi: 186, btch: 31 usd: 162 2013-02-21 13:08:37 CPU 7: hi: 186, btch: 31 usd: 170 2013-02-21 13:08:37 Node 1 Normal per-cpu: 2013-02-21 13:08:37 CPU 0: hi: 186, btch: 31 usd: 168 2013-02-21 13:08:37 CPU 1: hi: 186, btch: 31 usd: 169 2013-02-21 13:08:38 CPU 2: hi: 186, btch: 31 usd: 150 2013-02-21 13:08:38 CPU 3: hi: 186, btch: 31 usd: 176 2013-02-21 13:08:38 CPU 4: hi: 186, btch: 31 usd: 38 2013-02-21 13:08:38 CPU 5: hi: 186, btch: 31 usd: 57 2013-02-21 13:08:38 CPU 6: hi: 186, btch: 31 usd: 14 2013-02-21 13:08:38 CPU 7: hi: 186, btch: 31 usd: 71 2013-02-21 13:08:38 active_anon:1263170 inactive_anon:193369 isolated_anon:0 2013-02-21 13:08:38 active_file:242531 inactive_file:833544 isolated_file:0 2013-02-21 13:08:38 unevictable:67 dirty:13 writeback:5 unstable:0 2013-02-21 13:08:38 free:5230784 slab_reclaimable:69311 slab_unreclaimable:241977 2013-02-21 13:08:38 mapped:59848 shmem:1 pagetables:4602 bounce:0 2013-02-21 13:08:38 Node 0 DMA free:15680kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15296kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2013-02-21 13:08:38 lowmem_reserve[]: 0 3511 16136 16136 2013-02-21 13:08:39 Node 0 DMA32 free:1212680kB min:9792kB low:12240kB high:14688kB active_anon:1559344kB inactive_anon:403800kB active_file:1976kB inactive_file:15572kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3596192kB mlocked:0kB dirty:0kB writeback:0kB mapped:440kB shmem:0kB slab_reclaimable:1748kB slab_unreclaimable:12192kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2013-02-21 13:08:39 lowmem_reserve[]: 0 0 12625 12625 2013-02-21 13:08:39 Node 0 Normal free:9179492kB min:35208kB low:44008kB high:52812kB active_anon:901316kB inactive_anon:146612kB active_file:563048kB inactive_file:1441824kB unevictable:228kB isolated(anon):0kB isolated(file):0kB present:12928000kB mlocked:228kB dirty:28kB writeback:8kB mapped:118484kB shmem:4kB slab_reclaimable:137496kB slab_unreclaimable:449144kB kernel_stack:3040kB pagetables:6728kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2013-02-21 13:08:39 lowmem_reserve[]: 0 0 0 0 2013-02-21 13:08:40 Node 1 Normal free:10514920kB min:45064kB low:56328kB high:67596kB active_anon:2592220kB inactive_anon:223064kB active_file:405100kB inactive_file:1876780kB unevictable:40kB isolated(anon):0kB isolated(file):0kB present:16547840kB mlocked:40kB dirty:32kB writeback:12kB mapped:120468kB shmem:0kB slab_reclaimable:137996kB slab_unreclaimable:506544kB kernel_stack:504kB pagetables:11680kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 2013-02-21 13:08:40 lowmem_reserve[]: 0 0 0 0 2013-02-21 13:08:40 Node 0 DMA: 2*4kB 1*8kB 1*16kB 1*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15680kB 2013-02-21 13:08:40 Node 0 DMA32: 1426*4kB 1066*8kB 727*16kB 526*32kB 409*64kB 342*128kB 269*256kB 170*512kB 96*1024kB 1*2048kB 206*4096kB = 1212680kB 2013-02-21 13:08:40 Node 0 Normal: 62901*4kB 74438*8kB 39816*16kB 19763*32kB 10988*64kB 7105*128kB 5082*256kB 3474*512kB 2315*1024kB 0*2048kB 0*4096kB = 9179492kB 2013-02-21 13:08:40 Node 1 Normal: 82125*4kB 79959*8kB 53323*16kB 23474*32kB 14262*64kB 8254*128kB 5838*256kB 4352*512kB 2198*1024kB 0*2048kB 0*4096kB = 10515292kB 2013-02-21 13:08:41 1077318 total pagecache pages 2013-02-21 13:08:41 1081 pages in swap cache 2013-02-21 13:08:41 Swap cache stats: add 5473017, delete 5471936, find 5287546/5404444 2013-02-21 13:08:41 Free swap = 16749628kB 2013-02-21 13:08:41 Total swap = 16777208kB 2013-02-21 13:08:41 8388607 pages RAM 2013-02-21 13:08:41 168921 pages reserved 2013-02-21 13:08:41 1208772 pages shared 2013-02-21 13:08:41 1936948 pages non-shared 2013-02-21 13:08:41 [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name 2013-02-21 13:08:41 [ 555] 0 555 4378 82 5 -17 -1000 udevd 2013-02-21 13:08:41 [ 1784] 0 1784 23293 111 0 -17 -1000 auditd 2013-02-21 13:08:41 [ 1802] 0 1802 1539 7 4 0 0 portreserve 2013-02-21 13:08:41 [ 1870] 0 1870 2286 88 4 0 0 irqbalance 2013-02-21 13:08:42 [ 1889] 32 1889 4743 111 0 0 0 rpcbind 2013-02-21 13:08:42 [ 1978] 81 1978 6998 141 6 0 0 dbus-daemon 2013-02-21 13:08:42 [ 1996] 29 1996 5836 52 0 0 0 rpc.statd 2013-02-21 13:08:42 [ 2041] 0 2041 7354 132 2 0 0 rpc.idmapd 2013-02-21 13:08:42 [ 2051] 0 2051 48868 7 4 0 0 cupsd 2013-02-21 13:08:42 [ 2076] 0 2076 1019 6 3 0 0 acpid 2013-02-21 13:08:42 [ 2095] 0 2095 54399 91 2 0 0 ypbind 2013-02-21 13:08:42 [ 2123] 0 2123 95622 749 0 0 0 automount 2013-02-21 13:08:42 [ 2139] 28 2139 216410 348 0 0 0 nscd 2013-02-21 13:08:42 [ 2162] 0 2162 2165 104 1 0 0 afsd 2013-02-21 13:08:42 [ 2191] 0 2191 15635 86 7 -17 -1000 sshd 2013-02-21 13:08:42 [ 2210] 38 2210 6681 147 1 0 0 ntpd 2013-02-21 13:08:42 [ 2286] 0 2286 19859 65 0 0 0 master 2013-02-21 13:08:43 [ 2293] 89 2293 19922 151 1 0 0 qmgr 2013-02-21 13:08:43 [ 2296] 0 2296 27016 50 6 0 0 abrt-dump-oops 2013-02-21 13:08:43 [ 2305] 0 2305 27158 7 0 0 0 abrtd 2013-02-21 13:08:43 [ 2313] 8440 2313 9410 23 4 0 0 nrpe 2013-02-21 13:08:43 [ 2321] 0 2321 28918 62 0 0 0 crond 2013-02-21 13:08:43 [ 2337] 2 2337 2198 143 4 0 0 snapserv 2013-02-21 13:08:43 [ 2605] 0 2605 6146 204 1 0 0 lim 2013-02-21 13:08:43 [ 2630] 0 2630 5792 1228 1 0 0 res 2013-02-21 13:08:43 [ 2655] 0 2655 7035 1515 1 0 0 sbatchd 2013-02-21 13:08:43 [ 2675] 0 2675 5033 15 5 0 0 atd 2013-02-21 13:08:43 [ 2695] 8119 2695 4489 1220 0 0 0 gmond 2013-02-21 13:08:43 [ 2736] 0 2736 14286 16 4 0 0 certmonger 2013-02-21 13:08:43 [ 2762] 8119 2762 1721 172 0 0 0 gangliaCPUthrot 2013-02-21 13:08:44 [ 2809] 0 2809 1018 7 1 0 0 agetty 2013-02-21 13:08:44 [ 2811] 0 2811 1015 7 1 0 0 mingetty 2013-02-21 13:08:44 [ 2813] 0 2813 1015 7 3 0 0 mingetty 2013-02-21 13:08:44 [ 2815] 0 2815 1015 7 7 0 0 mingetty 2013-02-21 13:08:44 [ 2817] 0 2817 1015 7 5 0 0 mingetty 2013-02-21 13:08:44 [ 2819] 0 2819 1015 7 0 0 0 mingetty 2013-02-21 13:08:44 [ 2821] 0 2821 1015 7 6 0 0 mingetty 2013-02-21 13:08:44 [ 2823] 0 2823 4378 122 4 -17 -1000 udevd 2013-02-21 13:08:44 [ 2824] 0 2824 4378 6 2 -17 -1000 udevd 2013-02-21 13:08:44 [ 2825] 0 2825 3330 84 4 0 0 melim 2013-02-21 13:08:44 [ 2826] 0 2826 4071 200 5 0 0 pim 2013-02-21 13:08:44 [ 2827] 0 2827 27172 154 7 0 0 elim.hpc 2013-02-21 13:08:44 [ 2828] 0 2828 27172 155 4 0 0 elim.scratch 2013-02-21 13:08:45 [ 3725] 89 3725 19878 112 6 0 0 tlsmgr 2013-02-21 13:08:45 [11920] 10089 11920 2886 53 1 0 0 eauth 2013-02-21 13:08:45 [ 3906] 0 3906 63380 522 4 0 0 rsyslogd 2013-02-21 13:08:45 [28663] 0 28663 5372 190 1 0 0 ipmievd 2013-02-21 13:08:45 [15933] 9352 15933 4579 480 2 0 0 res 2013-02-21 13:08:45 [15951] 9018 15951 31419 557 0 0 0 TRClient 2013-02-21 13:08:45 [15952] 9352 15952 27028 339 0 0 0 1361473166.5684 2013-02-21 13:08:45 [15953] 9352 15953 1528 488 0 0 0 perl 2013-02-21 13:08:45 [15954] 9352 15954 1528 317 1 0 0 perl 2013-02-21 13:08:45 [15955] 9352 15955 13386 11939 7 0 0 strace 2013-02-21 13:08:45 [15959] 9352 15959 156218 153253 4 0 0 rockstar 2013-02-21 13:08:45 [15960] 9352 15960 28951 18957 5 0 0 rockstar 2013-02-21 13:08:45 [15961] 9352 15961 3162 368 5 0 0 rockstar 2013-02-21 13:08:46 [15962] 9352 15962 35247 23053 6 0 0 rockstar 2013-02-21 13:08:46 [15963] 9352 15963 19513 12813 6 0 0 rockstar 2013-02-21 13:08:46 [15964] 9352 15964 3084 290 5 0 0 rockstar 2013-02-21 13:08:46 [15965] 9352 15965 3260 466 5 0 0 rockstar 2013-02-21 13:08:46 [15966] 9352 15966 3320 526 6 0 0 rockstar 2013-02-21 13:08:46 [15967] 9352 15967 3515 721 5 0 0 rockstar 2013-02-21 13:08:46 [15968] 9352 15968 150889 147846 7 0 0 rockstar 2013-02-21 13:08:46 [15969] 9352 15969 151684 148672 4 0 0 rockstar 2013-02-21 13:08:46 [15970] 9352 15970 150837 147796 3 0 0 rockstar 2013-02-21 13:08:46 [15971] 9352 15971 158571 155558 3 0 0 rockstar 2013-02-21 13:08:46 [15972] 9352 15972 156209 153196 7 0 0 rockstar 2013-02-21 13:08:46 [15973] 9352 15973 152140 149125 2 0 0 rockstar 2013-02-21 13:08:46 [15974] 9352 15974 168502 160301 3 0 0 rockstar 2013-02-21 13:08:47 [27257] 89 27257 19879 809 2 0 0 pickup 2013-02-21 13:08:47 [ 9818] 0 9818 25225 142 7 0 0 sleep 2013-02-21 13:08:47 [15069] 0 15069 25225 142 4 0 0 sleep 2013-02-21 13:08:47 [16433] 9352 16433 57849 21545 3 0 0 rockstar 2013-02-21 13:08:47 [16434] 9352 16434 406930 65809 2 0 0 rockstar 2013-02-21 13:08:47 [16435] 9352 16435 57535 20224 4 0 0 rockstar 2013-02-21 13:08:47 [16436] 9352 16436 94268 32368 7 0 0 rockstar 2013-02-21 13:08:47 [16437] 9352 16437 337577 61542 3 0 0 rockstar 2013-02-21 13:08:47 [16438] 9352 16438 353996 57664 2 0 0 rockstar 2013-02-21 13:08:47 [16439] 9352 16439 430765 73552 5 0 0 rockstar 2013-02-21 13:08:47 [16440] 9352 16440 371354 56410 0 0 0 rockstar 2013-02-21 13:08:47 Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled