Lustre: DEBUG MARKER: /usr/sbin/lctl mark == performance-sanity test 7: getattr small file ======== 22:41:52 \(1369546912\) Lustre: DEBUG MARKER: == performance-sanity test 7: getattr small file ======== 22:41:52 (1369546912) Lustre: DEBUG MARKER: /usr/sbin/lctl mark ===== mdsrate-stat-small.sh ====== Lustre: DEBUG MARKER: ===== mdsrate-stat-small.sh ====== Lustre: DEBUG MARKER: running=$(grep -c /mnt/mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi Lustre: DEBUG MARKER: running=$(grep -c /mnt/mds1' ' /proc/mounts); mpts=$(mount | grep -c /mnt/mds1' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi Lustre: DEBUG MARKER: mkdir -p /mnt/mds1 Lustre: DEBUG MARKER: mkdir -p /mnt/mds1; mount -t lustre -o loop,user_xattr,acl /dev/lvm-MDS/P1 /mnt/mds1 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: Lustre: lustre-MDT0000: used disk, loading LustreError: 11-0: lustre-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. LustreError: Skipped 2 previous similar messages Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us LNet: 648:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. LNet: 648:0:(debug.c:324:libcfs_debug_str2mask()) Skipped 1 previous similar message Lustre: DEBUG MARKER: e2label /dev/lvm-MDS/P1 Lustre: 12889:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1369546915/real 1369546915] req@ffff880012003000 x1436032537184368/t0(0) o8->lustre-OST0006-osc-MDT0000@10.10.4.151@tcp:28/4 lens 400/544 e 0 to 1 dl 1369546920 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 11-0: lustre-OST0005-osc-MDT0000: Communicating with 10.10.4.151@tcp, operation ost_connect failed with -19. Lustre: DEBUG MARKER: lctl get_param -n timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug3 Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug3 Lustre: DEBUG MARKER: /usr/sbin/lctl mark ===== mdsrate-stat-small.sh Test preparation: creating 1000000 files. Lustre: DEBUG MARKER: ===== mdsrate-stat-small.sh Test preparation: creating 1000000 files. Lustre: DEBUG MARKER: /usr/sbin/lctl mark ===== mdsrate-stat-small.sh ### 1 NODE STAT ### Lustre: DEBUG MARKER: ===== mdsrate-stat-small.sh Lustre: DEBUG MARKER: /usr/sbin/lctl mark ===== mdsrate-stat-small.sh ### 2 NODES STAT ### Lustre: DEBUG MARKER: ===== mdsrate-stat-small.sh Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mdsrate-stat-small mdsrate-stat-small.sh: test complete, cleaning up == 23:13:15 \(1369548795\) Lustre: DEBUG MARKER: == mdsrate-stat-small mdsrate-stat-small.sh: test complete, cleaning up == 23:13:15 (1369548795) sendmail invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 sendmail cpuset=/ mems_allowed=0 Pid: 1639, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? perf_event_task_sched_out+0x33/0x80 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? __mem_cgroup_uncharge_common+0x81/0x300 [] ? mem_cgroup_uncharge_swapcache+0x2b/0x60 [] ? cpumask_any_but+0x31/0x50 [] ? flush_tlb_page+0x48/0xb0 [] ? ptep_set_access_flags+0x6d/0x70 [] ? swap_cgroup_record+0xa3/0xc0 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_unrolled+0x86/0xb0 [] ? copy_user_generic+0xe/0x20 [] ? set_fd_set+0x49/0x60 [] ? core_sys_select+0x1bc/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? sys_select+0x47/0x110 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 30 active_anon:0 inactive_anon:3 isolated_anon:0 active_file:208092 inactive_file:207909 isolated_file:480 unevictable:0 dirty:459 writeback:1 unstable:0 free:13241 slab_reclaimable:13176 slab_unreclaimable:10179 mapped:1 shmem:0 pagetables:787 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44620kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:828620kB inactive_file:828284kB unevictable:0kB isolated(anon):0kB isolated(file):1920kB present:2052064kB mlocked:0kB dirty:1836kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52460kB slab_unreclaimable:40700kB kernel_stack:1496kB pagetables:3148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1506607 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9623*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44620kB 69416 total pagecache pages 3 pages in swap cache Swap cache stats: add 33445, delete 33442, find 17374/22783 Free swap = 4105544kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69933 pages shared 392724 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1147] 0 1147 62367 1 0 0 0 rsyslogd [ 1189] 32 1189 4743 1 0 0 0 rpcbind [ 1201] 0 1201 49811 1 0 0 0 sssd [ 1202] 0 1202 57001 1 0 0 0 sssd_be [ 1203] 0 1203 50720 1 0 0 0 sssd_nss [ 1204] 0 1204 48273 1 0 0 0 sssd_pam [ 1221] 29 1221 6355 1 0 0 0 rpc.statd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1147 (rsyslogd) score 1 or sacrifice child Killed process 1147, UID 0, (rsyslogd) total-vm:249468kB, anon-rss:0kB, file-rss:4kB Lustre: 476:0:(service.c:1889:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880037b59000 x1436074024325331/t0(0) o400->4cdcd8f7-da74-e70d-a566-17cc3f711c5f@10.10.4.153@tcp:0/0 lens 192/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 rs:main Q:Reg invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 rs:main Q:Reg cpuset=/ mems_allowed=0 Pid: 1148, comm: rs:main Q:Reg Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? __wake_up_bit+0x31/0x40 [] ? wake_up_bit+0x2f/0x40 [] ? wake_bit_function+0x0/0x50 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? perf_event_task_sched_out+0x33/0x80 [] ? dequeue_entity+0x113/0x2e0 [] ? __dequeue_entity+0x30/0x50 [] ? __switch_to+0x26e/0x320 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? __get_user_8+0x1c/0x23 [] ? exit_robust_list+0x5d/0x160 [] ? __delayacct_add_tsk+0x196/0x1b0 [] ? mm_release+0x113/0x130 [] ? exit_mm+0x26/0x180 [] ? do_exit+0x15f/0x870 [] ? do_group_exit+0x58/0xd0 [] ? get_signal_to_deliver+0x1f6/0x460 [] ? do_signal+0x75/0x800 [] ? sys_futex+0x7b/0x170 [] ? do_notify_resume+0x90/0xc0 [] ? int_signal+0x12/0x17 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 55 active_anon:0 inactive_anon:22 isolated_anon:0 active_file:208263 inactive_file:207769 isolated_file:384 unevictable:0 dirty:347 writeback:1 unstable:0 free:13240 slab_reclaimable:13205 slab_unreclaimable:10185 mapped:5 shmem:0 pagetables:787 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44616kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:76kB active_file:829304kB inactive_file:827724kB unevictable:0kB isolated(anon):0kB isolated(file):1536kB present:2052064kB mlocked:0kB dirty:1388kB writeback:4kB mapped:20kB shmem:0kB slab_reclaimable:52576kB slab_unreclaimable:40724kB kernel_stack:1480kB pagetables:3148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:409293 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9622*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44616kB 69355 total pagecache pages 10 pages in swap cache Swap cache stats: add 33511, delete 33501, find 17398/22823 Free swap = 4105592kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69741 pages shared 392885 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 2 0 -17 -1000 auditd [ 1148] 0 1147 62367 2 0 0 0 rs:main Q:Reg [ 1189] 32 1189 4743 1 0 0 0 rpcbind [ 1201] 0 1201 49811 3 0 0 0 sssd [ 1202] 0 1202 57001 1 0 0 0 sssd_be [ 1203] 0 1203 50720 1 0 0 0 sssd_nss [ 1204] 0 1204 48273 1 0 0 0 sssd_pam [ 1221] 29 1221 6355 1 0 0 0 rpc.statd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 17 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty sendmail invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 sendmail cpuset=/ mems_allowed=0 Pid: 1639, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? perf_event_task_sched_out+0x33/0x80 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? __mem_cgroup_uncharge_common+0x81/0x300 [] ? mem_cgroup_uncharge_swapcache+0x2b/0x60 [] ? cpumask_any_but+0x31/0x50 [] ? flush_tlb_page+0x48/0xb0 [] ? ptep_set_access_flags+0x6d/0x70 [] ? swap_cgroup_record+0xa3/0xc0 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_unrolled+0x86/0xb0 [] ? copy_user_generic+0xe/0x20 [] ? set_fd_set+0x49/0x60 [] ? core_sys_select+0x1bc/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? sys_select+0x47/0x110 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 166 active_anon:0 inactive_anon:16 isolated_anon:0 active_file:208062 inactive_file:207978 isolated_file:192 unevictable:0 dirty:0 writeback:0 unstable:0 free:13438 slab_reclaimable:13133 slab_unreclaimable:10170 mapped:27 shmem:0 pagetables:766 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:45408kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:52kB active_file:828500kB inactive_file:828560kB unevictable:0kB isolated(anon):0kB isolated(file):768kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:108kB shmem:0kB slab_reclaimable:52288kB slab_unreclaimable:40664kB kernel_stack:1464kB pagetables:3064kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2097341 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9820*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 45408kB 69177 total pagecache pages 11 pages in swap cache Swap cache stats: add 35611, delete 35600, find 18296/24256 Free swap = 4108912kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69390 pages shared 392902 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1189] 32 1189 4743 1 0 0 0 rpcbind [ 1201] 0 1201 49811 38 0 0 0 sssd [ 1202] 0 1202 57001 1 0 0 0 sssd_be [ 1203] 0 1203 50720 1 0 0 0 sssd_nss [ 1204] 0 1204 48273 1 0 0 0 sssd_pam [ 1221] 29 1221 6355 1 0 0 0 rpc.statd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1189 (rpcbind) score 1 or sacrifice child Killed process 1189, UID 32, (rpcbind) total-vm:18972kB, anon-rss:0kB, file-rss:4kB sendmail invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 sendmail cpuset=/ mems_allowed=0 Pid: 1639, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? perf_event_task_sched_out+0x33/0x80 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? __mem_cgroup_uncharge_common+0x81/0x300 [] ? mem_cgroup_uncharge_swapcache+0x2b/0x60 [] ? cpumask_any_but+0x31/0x50 [] ? flush_tlb_page+0x48/0xb0 [] ? ptep_set_access_flags+0x6d/0x70 [] ? swap_cgroup_record+0xa3/0xc0 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_unrolled+0x86/0xb0 [] ? copy_user_generic+0xe/0x20 [] ? set_fd_set+0x49/0x60 [] ? core_sys_select+0x1bc/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? sys_select+0x47/0x110 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 17 active_anon:0 inactive_anon:8 isolated_anon:0 active_file:218903 inactive_file:197406 isolated_file:256 unevictable:0 dirty:341 writeback:1 unstable:0 free:13187 slab_reclaimable:13152 slab_unreclaimable:10238 mapped:1 shmem:0 pagetables:752 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44404kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:20kB active_file:871864kB inactive_file:786272kB unevictable:0kB isolated(anon):0kB isolated(file):1024kB present:2052064kB mlocked:0kB dirty:1364kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52364kB slab_unreclaimable:40936kB kernel_stack:1456kB pagetables:3008kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4028354 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9559*4kB 7*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44404kB 69499 total pagecache pages 3 pages in swap cache Swap cache stats: add 35684, delete 35681, find 18346/24327 Free swap = 4109168kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69781 pages shared 392922 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1201] 0 1201 49811 1 0 0 0 sssd [ 1202] 0 1202 57001 1 0 0 0 sssd_be [ 1203] 0 1203 50720 1 0 0 0 sssd_nss [ 1204] 0 1204 48273 1 0 0 0 sssd_pam [ 1221] 29 1221 6355 1 0 0 0 rpc.statd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1201 (sssd) score 1 or sacrifice child Killed process 1202, UID 0, (sssd_be) total-vm:228004kB, anon-rss:0kB, file-rss:4kB sendmail invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 sendmail cpuset=/ mems_allowed=0 Pid: 1639, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? perf_event_task_sched_out+0x33/0x80 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? __mem_cgroup_uncharge_common+0x81/0x300 [] ? mem_cgroup_uncharge_swapcache+0x2b/0x60 [] ? cpumask_any_but+0x31/0x50 [] ? flush_tlb_page+0x48/0xb0 [] ? ptep_set_access_flags+0x6d/0x70 [] ? swap_cgroup_record+0xa3/0xc0 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_unrolled+0x86/0xb0 [] ? copy_user_generic+0xe/0x20 [] ? set_fd_set+0x49/0x60 [] ? core_sys_select+0x1bc/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? sys_select+0x47/0x110 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 177 active_anon:0 inactive_anon:12 isolated_anon:0 active_file:208008 inactive_file:207965 isolated_file:288 unevictable:0 dirty:40 writeback:1 unstable:0 free:13265 slab_reclaimable:13154 slab_unreclaimable:10392 mapped:1 shmem:0 pagetables:681 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44716kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:36kB active_file:828284kB inactive_file:828508kB unevictable:0kB isolated(anon):0kB isolated(file):1152kB present:2052064kB mlocked:0kB dirty:160kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52372kB slab_unreclaimable:41552kB kernel_stack:1456kB pagetables:2724kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3620158 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9647*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44716kB 69206 total pagecache pages 3 pages in swap cache Swap cache stats: add 35695, delete 35692, find 18347/24331 Free swap = 4110872kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69507 pages shared 392968 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1201] 0 1201 49811 1 0 0 0 sssd [ 1203] 0 1203 50720 1 0 0 0 sssd_nss [ 1204] 0 1204 48273 1 0 0 0 sssd_pam [ 1221] 29 1221 6355 1 0 0 0 rpc.statd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1201 (sssd) score 1 or sacrifice child Killed process 1203, UID 0, (sssd_nss) total-vm:202880kB, anon-rss:0kB, file-rss:4kB sendmail invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 sendmail cpuset=/ mems_allowed=0 Pid: 1639, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? perf_event_task_sched_out+0x33/0x80 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? __mem_cgroup_uncharge_common+0x81/0x300 [] ? mem_cgroup_uncharge_swapcache+0x2b/0x60 [] ? cpumask_any_but+0x31/0x50 [] ? flush_tlb_page+0x48/0xb0 [] ? ptep_set_access_flags+0x6d/0x70 [] ? swap_cgroup_record+0xa3/0xc0 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_unrolled+0x86/0xb0 [] ? copy_user_generic+0xe/0x20 [] ? set_fd_set+0x49/0x60 [] ? core_sys_select+0x1bc/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? sys_select+0x47/0x110 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 6 active_anon:0 inactive_anon:4 isolated_anon:0 active_file:208012 inactive_file:207946 isolated_file:256 unevictable:0 dirty:0 writeback:0 unstable:0 free:13785 slab_reclaimable:13152 slab_unreclaimable:10152 mapped:1 shmem:0 pagetables:625 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:46796kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:4kB active_file:828300kB inactive_file:828432kB unevictable:0kB isolated(anon):0kB isolated(file):1024kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:52364kB slab_unreclaimable:40592kB kernel_stack:1456kB pagetables:2500kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2014170 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 10027*4kB 22*8kB 15*16kB 8*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 46796kB 69150 total pagecache pages 4 pages in swap cache Swap cache stats: add 35699, delete 35695, find 18348/24333 Free swap = 4111548kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69425 pages shared 392673 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1201] 0 1201 49811 1 0 0 0 sssd [ 1204] 0 1204 48273 1 0 0 0 sssd_pam [ 1221] 29 1221 6355 1 0 0 0 rpc.statd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1201 (sssd) score 1 or sacrifice child Killed process 1204, UID 0, (sssd_pam) total-vm:193092kB, anon-rss:0kB, file-rss:4kB crond invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0 crond cpuset=/ mems_allowed=0 Pid: 1669, comm: crond Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:6 isolated_anon:0 active_file:208146 inactive_file:208057 isolated_file:448 unevictable:0 dirty:69 writeback:1 unstable:0 free:13454 slab_reclaimable:13128 slab_unreclaimable:10138 mapped:1 shmem:0 pagetables:570 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:45472kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:12kB active_file:828836kB inactive_file:828876kB unevictable:0kB isolated(anon):0kB isolated(file):1792kB present:2052064kB mlocked:0kB dirty:276kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52268kB slab_unreclaimable:40536kB kernel_stack:1432kB pagetables:2280kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9832*4kB 4*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 45472kB 69590 total pagecache pages 3 pages in swap cache Swap cache stats: add 36685, delete 36682, find 18764/24956 Free swap = 4112100kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70062 pages shared 392439 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1201] 0 1201 49811 1 0 0 0 sssd [ 1221] 29 1221 6355 1 0 0 0 rpc.statd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 3 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1201 (sssd) score 1 or sacrifice child Killed process 1201, UID 0, (sssd) total-vm:199244kB, anon-rss:0kB, file-rss:4kB sssd: page allocation failure. order:1, mode:0xd0 Pid: 1201, comm: sssd Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? __alloc_pages_nodemask+0x757/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? getnstimeofday+0x60/0xf0 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:6 isolated_anon:0 active_file:208291 inactive_file:208209 isolated_file:288 unevictable:0 dirty:212 writeback:1 unstable:0 free:13300 slab_reclaimable:13138 slab_unreclaimable:10143 mapped:1 shmem:0 pagetables:570 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44856kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:12kB active_file:829416kB inactive_file:829484kB unevictable:0kB isolated(anon):0kB isolated(file):1152kB present:2052064kB mlocked:0kB dirty:848kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52308kB slab_unreclaimable:40556kB kernel_stack:1432kB pagetables:2280kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:800178 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9681*4kB 4*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44868kB 69726 total pagecache pages 6 pages in swap cache Swap cache stats: add 36688, delete 36682, find 18764/24956 Free swap = 4112088kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70043 pages shared 392593 pages non-shared sendmail invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 sendmail cpuset=/ mems_allowed=0 Pid: 1639, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __page_cache_alloc+0x87/0x90 [] ? find_get_page+0x1e/0xa0 [] ? filemap_fault+0x1a7/0x500 [] ? __do_fault+0x54/0x530 [] ? mem_cgroup_uncharge_swap+0x27/0x90 [] ? handle_pte_fault+0xf7/0xb50 [] ? copy_user_generic+0xe/0x20 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? poll_select_copy_remaining+0xf8/0x150 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 19 active_anon:0 inactive_anon:6 isolated_anon:0 active_file:208332 inactive_file:208246 isolated_file:224 unevictable:0 dirty:1 writeback:209 unstable:0 free:13315 slab_reclaimable:13138 slab_unreclaimable:10143 mapped:1 shmem:0 pagetables:522 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44916kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:12kB active_file:829580kB inactive_file:829632kB unevictable:0kB isolated(anon):0kB isolated(file):896kB present:2052064kB mlocked:0kB dirty:4kB writeback:836kB mapped:4kB shmem:0kB slab_reclaimable:52308kB slab_unreclaimable:40556kB kernel_stack:1432kB pagetables:2088kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:354511 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9651*4kB 15*8kB 7*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44916kB 69742 total pagecache pages 6 pages in swap cache Swap cache stats: add 36696, delete 36690, find 18765/24959 Free swap = 4112704kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69978 pages shared 392569 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1221] 29 1221 6355 1 0 0 0 rpc.statd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1221 (rpc.statd) score 1 or sacrifice child Killed process 1221, UID 29, (rpc.statd) total-vm:25420kB, anon-rss:0kB, file-rss:4kB ll_mgs_0000 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 ll_mgs_0000 cpuset=/ mems_allowed=0 Pid: 469, comm: ll_mgs_0000 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:3 isolated_anon:0 active_file:208343 inactive_file:208245 isolated_file:288 unevictable:0 dirty:275 writeback:2 unstable:0 free:13297 slab_reclaimable:13170 slab_unreclaimable:10116 mapped:1 shmem:0 pagetables:498 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44844kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:829624kB inactive_file:829628kB unevictable:0kB isolated(anon):0kB isolated(file):1152kB present:2052064kB mlocked:0kB dirty:1100kB writeback:8kB mapped:4kB shmem:0kB slab_reclaimable:52436kB slab_unreclaimable:40448kB kernel_stack:1424kB pagetables:1992kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9611*4kB 14*8kB 9*16kB 4*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44844kB 69811 total pagecache pages 3 pages in swap cache Swap cache stats: add 36813, delete 36810, find 18808/25024 Free swap = 4113168kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70129 pages shared 392477 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1347] 81 1347 5869 1 0 0 0 dbus-daemon [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1347 (dbus-daemon) score 1 or sacrifice child Killed process 1347, UID 81, (dbus-daemon) total-vm:23476kB, anon-rss:0kB, file-rss:4kB crond invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0 crond cpuset=/ mems_allowed=0 Pid: 1669, comm: crond Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:3 isolated_anon:0 active_file:211168 inactive_file:205353 isolated_file:384 unevictable:0 dirty:300 writeback:1 unstable:0 free:13292 slab_reclaimable:13173 slab_unreclaimable:10108 mapped:1 shmem:0 pagetables:483 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44824kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:840924kB inactive_file:818060kB unevictable:0kB isolated(anon):0kB isolated(file):1536kB present:2052064kB mlocked:0kB dirty:1200kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52448kB slab_unreclaimable:40416kB kernel_stack:1424kB pagetables:1932kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9598*4kB 20*8kB 8*16kB 4*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44824kB 69836 total pagecache pages 3 pages in swap cache Swap cache stats: add 36829, delete 36826, find 18809/25028 Free swap = 4113436kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70250 pages shared 392390 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1386] 0 1386 1019 0 0 0 0 acpid [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1386 (acpid) score 1 or sacrifice child Killed process 1386, UID 0, (acpid) total-vm:4076kB, anon-rss:0kB, file-rss:0kB crond invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0 crond cpuset=/ mems_allowed=0 Pid: 1669, comm: crond Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:3 isolated_anon:0 active_file:251730 inactive_file:164718 isolated_file:480 unevictable:0 dirty:327 writeback:1 unstable:0 free:11538 slab_reclaimable:13173 slab_unreclaimable:11836 mapped:1 shmem:0 pagetables:477 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:37808kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:1003172kB inactive_file:655520kB unevictable:0kB isolated(anon):0kB isolated(file):1920kB present:2052064kB mlocked:0kB dirty:1308kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52448kB slab_unreclaimable:47328kB kernel_stack:1424kB pagetables:1908kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 7845*4kB 20*8kB 8*16kB 4*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 37812kB 69863 total pagecache pages 3 pages in swap cache Swap cache stats: add 36836, delete 36833, find 18809/25029 Free swap = 4113548kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70372 pages shared 394043 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1395] 68 1395 6794 1 0 0 0 hald [ 1396] 0 1396 4525 1 0 0 0 hald-runner [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1395 (hald) score 1 or sacrifice child Killed process 1396, UID 0, (hald-runner) total-vm:18100kB, anon-rss:0kB, file-rss:4kB mdt00_004 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 mdt00_004 cpuset=/ mems_allowed=0 Pid: 1114, comm: mdt00_004 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:3 isolated_anon:0 active_file:246542 inactive_file:169719 isolated_file:608 unevictable:0 dirty:269 writeback:1 unstable:0 free:11193 slab_reclaimable:13172 slab_unreclaimable:12262 mapped:1 shmem:0 pagetables:462 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:36428kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:982420kB inactive_file:675524kB unevictable:0kB isolated(anon):0kB isolated(file):2432kB present:2052064kB mlocked:0kB dirty:1076kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52444kB slab_unreclaimable:49032kB kernel_stack:1424kB pagetables:1848kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 7489*4kB 23*8kB 9*16kB 4*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 36428kB 69805 total pagecache pages 3 pages in swap cache Swap cache stats: add 36852, delete 36849, find 18810/25032 Free swap = 4113756kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70440 pages shared 394349 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1395] 68 1395 6794 1 0 0 0 hald [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1395 (hald) score 1 or sacrifice child Killed process 1395, UID 68, (hald) total-vm:27176kB, anon-rss:0kB, file-rss:4kB crond invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0 crond cpuset=/ mems_allowed=0 Pid: 1669, comm: crond Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:3 isolated_anon:0 active_file:208016 inactive_file:207912 isolated_file:704 unevictable:0 dirty:28 writeback:1 unstable:0 free:11453 slab_reclaimable:13167 slab_unreclaimable:12276 mapped:1 shmem:0 pagetables:444 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:37468kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:828316kB inactive_file:828296kB unevictable:0kB isolated(anon):0kB isolated(file):2816kB present:2052064kB mlocked:0kB dirty:112kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52424kB slab_unreclaimable:49088kB kernel_stack:1424kB pagetables:1776kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 7725*4kB 31*8kB 11*16kB 4*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 37468kB 69567 total pagecache pages 3 pages in swap cache Swap cache stats: add 36856, delete 36853, find 18811/25035 Free swap = 4114464kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70294 pages shared 394224 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1424] 0 1424 5054 1 0 0 0 hald-addon-inpu [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1424 (hald-addon-inpu) score 1 or sacrifice child Killed process 1424, UID 0, (hald-addon-inpu) total-vm:20216kB, anon-rss:0kB, file-rss:4kB LustreError: 475:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.10.4.151@tcp: deadline 10:33s ago req@ffff88003abe7800 x1436032519866440/t0(0) o400->lustre-MDT0000-lwp-OST0006_UUID@10.10.4.151@tcp:0/0 lens 224/0 e 0 to 0 dl 1369549958 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 Lustre: 475:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (10:33s); client may timeout. req@ffff88003abe7800 x1436032519866440/t0(0) o400->lustre-MDT0000-lwp-OST0006_UUID@10.10.4.151@tcp:0/0 lens 224/0 e 0 to 0 dl 1369549958 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 Lustre: 475:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 3 previous similar messages LNet: Service thread pid 475 completed after 43.82s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 12890:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1369549958/real 0] req@ffff88005d1b4400 x1436032537207156/t0(0) o13->lustre-OST0004-osc-MDT0000@10.10.4.151@tcp:7/4 lens 224/368 e 0 to 1 dl 1369549965 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 12890:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: lustre-OST0004-osc-MDT0000: Connection to lustre-OST0004 (at 10.10.4.151@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 475:0:(service.c:1889:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 43s req@ffff880037ed6800 x1436074024331292/t0(0) o400->4cdcd8f7-da74-e70d-a566-17cc3f711c5f@10.10.4.153@tcp:0/0 lens 192/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 Lustre: mdt: This server is not able to keep up with request traffic (cpu-bound). Lustre: 475:0:(service.c:1500:ptlrpc_at_check_timed()) earlyQ=25 reqQ=0 recA=3, svcEst=1, delay=38980(jiff) Lustre: 475:0:(service.c:1301:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-10s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880062b6b000 x1436074024331273/t0(0) o36->4cdcd8f7-da74-e70d-a566-17cc3f711c5f@10.10.4.153@tcp:0/0 lens 456/696 e 0 to 0 dl 1369549982 ref 2 fl Interpret:/0/0 rc 0/0 Lustre: 475:0:(service.c:1301:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-24s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880037c4a000 x1436032537207152/t0(0) o400->lustre-MDT0000-lwp-MDT0000_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 1369549968 ref 2 fl New:H/0/ffffffff rc 0/-1 LustreError: 475:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.10.4.153@tcp: deadline 10:34s ago req@ffff880037ed6800 x1436074024331292/t0(0) o400->4cdcd8f7-da74-e70d-a566-17cc3f711c5f@10.10.4.153@tcp:0/0 lens 192/0 e 0 to 0 dl 1369549958 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 Lustre: MGS: Client b19567ff-2911-ef97-2838-c1ef1ffd043b (at 10.10.4.152@tcp) reconnecting Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-MDT0000_UUID (at 0@lo) reconnecting Lustre: lustre-OST0004-osc-MDT0000: Connection restored to lustre-OST0004 (at 10.10.4.151@tcp) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Client lustre-MDT0000-lwp-OST0004_UUID (at 10.10.4.151@tcp) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 1 previous similar message mdt00_004 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 mdt00_004 cpuset=/ mems_allowed=0 Pid: 1114, comm: mdt00_004 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:6 isolated_anon:0 active_file:208217 inactive_file:208045 isolated_file:352 unevictable:0 dirty:30 writeback:1 unstable:0 free:13641 slab_reclaimable:13184 slab_unreclaimable:10098 mapped:1 shmem:0 pagetables:428 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:46220kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:12kB active_file:829120kB inactive_file:828828kB unevictable:0kB isolated(anon):0kB isolated(file):1408kB present:2052064kB mlocked:0kB dirty:120kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52492kB slab_unreclaimable:40376kB kernel_stack:1424kB pagetables:1712kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9949*4kB 23*8kB 6*16kB 4*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 46220kB 69562 total pagecache pages 6 pages in swap cache Swap cache stats: add 36916, delete 36910, find 18824/25059 Free swap = 4114636kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69933 pages shared 392342 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1434] 68 1434 4450 1 0 0 0 hald-addon-acpi [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1434 (hald-addon-acpi) score 1 or sacrifice child Killed process 1434, UID 68, (hald-addon-acpi) total-vm:17800kB, anon-rss:0kB, file-rss:4kB Lustre: 12891:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1369549953/real 1369549953] req@ffff880012003000 x1436032537207116/t0(0) o400->lustre-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1369549964 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 12891:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages LNet: Service thread pid 700 completed after 46.65s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). mdt00_004 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 mdt00_004 cpuset=/ mems_allowed=0 Pid: 1114, comm: mdt00_004 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:5 isolated_anon:0 active_file:208248 inactive_file:208159 isolated_file:256 unevictable:0 dirty:68 writeback:1 unstable:0 free:13613 slab_reclaimable:13203 slab_unreclaimable:10100 mapped:1 shmem:0 pagetables:412 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:46108kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:8kB active_file:829244kB inactive_file:829284kB unevictable:0kB isolated(anon):0kB isolated(file):1024kB present:2052064kB mlocked:0kB dirty:272kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52568kB slab_unreclaimable:40384kB kernel_stack:1424kB pagetables:1648kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9879*4kB 44*8kB 10*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 46108kB 69600 total pagecache pages 5 pages in swap cache Swap cache stats: add 37088, delete 37083, find 18964/25241 Free swap = 4114800kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69874 pages shared 392447 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1454] 0 1454 168269 1 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1454 (automount) score 1 or sacrifice child Killed process 1454, UID 0, (automount) total-vm:673076kB, anon-rss:0kB, file-rss:4kB init invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 init cpuset=/ mems_allowed=0 Pid: 1, comm: init Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? sync_page_killable+0x0/0x40 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? handle_mm_fault+0x23a/0x310 [] ? copy_user_generic_unrolled+0x3e/0xb0 [] ? __do_page_fault+0x139/0x480 [] ? sys_waitid+0xa8/0x1f0 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 64 active_anon:0 inactive_anon:82 isolated_anon:0 active_file:208491 inactive_file:208123 isolated_file:256 unevictable:0 dirty:36 writeback:1 unstable:0 free:13251 slab_reclaimable:13203 slab_unreclaimable:10100 mapped:17 shmem:0 pagetables:412 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44660kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:316kB active_file:830216kB inactive_file:829140kB unevictable:0kB isolated(anon):0kB isolated(file):1024kB present:2052064kB mlocked:0kB dirty:144kB writeback:4kB mapped:68kB shmem:0kB slab_reclaimable:52568kB slab_unreclaimable:40384kB kernel_stack:1424kB pagetables:1648kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:563485 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9633*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44660kB 69888 total pagecache pages 87 pages in swap cache Swap cache stats: add 37174, delete 37087, find 18981/25272 Free swap = 4114816kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 69864 pages shared 392739 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1456] 0 1454 168269 7 0 0 0 automount [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 21 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1456 (automount) score 1 or sacrifice child Killed process 1456, UID 0, (automount) total-vm:673076kB, anon-rss:24kB, file-rss:4kB Lustre: lustre-MDT0000: Client d212be8a-cd9e-b654-7c84-9fe1868ce834 (at 10.10.4.152@tcp) reconnecting Lustre: Skipped 33 previous similar messages LNet: Service thread pid 476 completed after 65.33s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: lustre-MDT0000: Client d212be8a-cd9e-b654-7c84-9fe1868ce834 (at 10.10.4.152@tcp) reconnecting Lustre: Skipped 2 previous similar messages Lustre: 470:0:(service.c:1889:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880018bcf050 x1436074024331373/t0(0) o400->5f6d6d54-40b2-a585-f89f-dec0f4013e3a@10.10.4.153@tcp:0/0 lens 192/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 Lustre: 470:0:(service.c:1889:ptlrpc_server_handle_req_in()) Skipped 21 previous similar messages Lustre: mgs: This server is not able to keep up with request traffic (cpu-bound). Lustre: 470:0:(service.c:1500:ptlrpc_at_check_timed()) earlyQ=3 reqQ=0 recA=1, svcEst=9, delay=7937(jiff) automount: page allocation failure. order:1, mode:0xd0 Pid: 1456, comm: automount Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? __alloc_pages_nodemask+0x757/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? wake_up_new_task+0xd9/0x130 [] ? do_fork+0x94/0x460 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:83 isolated_anon:0 active_file:207312 inactive_file:207098 isolated_file:160 unevictable:0 dirty:0 writeback:11 unstable:0 free:16359 slab_reclaimable:13015 slab_unreclaimable:9587 mapped:109 shmem:0 pagetables:412 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:57092kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:320kB active_file:825500kB inactive_file:825040kB unevictable:0kB isolated(anon):0kB isolated(file):640kB present:2052064kB mlocked:0kB dirty:0kB writeback:44kB mapped:436kB shmem:0kB slab_reclaimable:51816kB slab_unreclaimable:38332kB kernel_stack:1328kB pagetables:1648kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 12623*4kB 57*8kB 4*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 57092kB 67572 total pagecache pages 60 pages in swap cache Swap cache stats: add 37513, delete 37453, find 19164/25548 Free swap = 4114892kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 67377 pages shared 392208 pages non-shared init: Disconnected from system bus crond invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0 crond cpuset=/ mems_allowed=0 Pid: 1669, comm: crond Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:4 isolated_anon:0 active_file:208553 inactive_file:208193 isolated_file:448 unevictable:0 dirty:221 writeback:1 unstable:0 free:10390 slab_reclaimable:13107 slab_unreclaimable:12993 mapped:1 shmem:0 pagetables:344 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:33216kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:4kB active_file:830464kB inactive_file:829420kB unevictable:0kB isolated(anon):0kB isolated(file):1792kB present:2052064kB mlocked:0kB dirty:884kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52184kB slab_unreclaimable:51956kB kernel_stack:1312kB pagetables:1376kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 6772*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 33216kB 70130 total pagecache pages 3 pages in swap cache Swap cache stats: add 37601, delete 37598, find 19194/25589 Free swap = 4119292kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70595 pages shared 394987 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1496] 0 1496 26826 0 0 0 0 rpc.rquotad [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 1 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1496 (rpc.rquotad) score 1 or sacrifice child Killed process 1496, UID 0, (rpc.rquotad) total-vm:107304kB, anon-rss:0kB, file-rss:0kB init invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 init cpuset=/ mems_allowed=0 Pid: 1, comm: init Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? handle_mm_fault+0x23a/0x310 [] ? pollwake+0x0/0x60 [] ? __do_page_fault+0x139/0x480 [] ? unlock_page+0x27/0x30 [] ? handle_pte_fault+0xf7/0xb50 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_unrolled+0x3e/0xb0 [] ? copy_user_generic+0xe/0x20 [] ? save_i387_xstate+0x1c7/0x220 [] ? do_signal+0x6f6/0x800 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? do_notify_resume+0x90/0xc0 [] ? int_signal+0x12/0x17 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 141 active_anon:0 inactive_anon:7 isolated_anon:0 active_file:208361 inactive_file:208216 isolated_file:384 unevictable:0 dirty:0 writeback:0 unstable:0 free:13463 slab_reclaimable:13103 slab_unreclaimable:10025 mapped:9 shmem:0 pagetables:335 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:45508kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:16kB active_file:829696kB inactive_file:829512kB unevictable:0kB isolated(anon):0kB isolated(file):1536kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:36kB shmem:0kB slab_reclaimable:52168kB slab_unreclaimable:40084kB kernel_stack:1312kB pagetables:1340kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3117990 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9845*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 45508kB 69910 total pagecache pages 6 pages in swap cache Swap cache stats: add 37622, delete 37616, find 19197/25598 Free swap = 4119432kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70297 pages shared 392032 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1500] 0 1500 5413 0 0 0 0 rpc.mountd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 4 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1500 (rpc.mountd) score 1 or sacrifice child Killed process 1500, UID 0, (rpc.mountd) total-vm:21652kB, anon-rss:0kB, file-rss:0kB init invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 init cpuset=/ mems_allowed=0 Pid: 1, comm: init Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? d_lookup+0x3c/0x60 [] ? handle_mm_fault+0x23a/0x310 [] ? call_rcu_sched+0x15/0x20 [] ? __do_page_fault+0x139/0x480 [] ? remove_wait_queue+0x3c/0x50 [] ? do_wait+0x17f/0x240 [] ? sys_waitid+0xa8/0x1f0 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 92 active_anon:0 inactive_anon:3 isolated_anon:0 active_file:208508 inactive_file:208443 isolated_file:288 unevictable:0 dirty:272 writeback:0 unstable:0 free:13240 slab_reclaimable:13115 slab_unreclaimable:10027 mapped:1 shmem:0 pagetables:320 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44616kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:830284kB inactive_file:830420kB unevictable:0kB isolated(anon):0kB isolated(file):1152kB present:2052064kB mlocked:0kB dirty:1088kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:52216kB slab_unreclaimable:40092kB kernel_stack:1304kB pagetables:1280kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2015396 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9622*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44616kB 70177 total pagecache pages 7 pages in swap cache Swap cache stats: add 37702, delete 37695, find 19215/25630 Free swap = 4120032kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70484 pages shared 392112 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1549] 0 1549 6290 1 0 0 0 rpc.idmapd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 7 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1549 (rpc.idmapd) score 1 or sacrifice child Killed process 1549, UID 0, (rpc.idmapd) total-vm:25160kB, anon-rss:0kB, file-rss:4kB ll_mgs_0000 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 ll_mgs_0000 cpuset=/ mems_allowed=0 Pid: 469, comm: ll_mgs_0000 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:6 inactive_anon:14 isolated_anon:0 active_file:208449 inactive_file:208280 isolated_file:256 unevictable:0 dirty:0 writeback:0 unstable:0 free:13606 slab_reclaimable:13124 slab_unreclaimable:10016 mapped:17 shmem:0 pagetables:304 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:46080kB min:44720kB low:55900kB high:67080kB active_anon:24kB inactive_anon:44kB active_file:830048kB inactive_file:829768kB unevictable:0kB isolated(anon):0kB isolated(file):1024kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:68kB shmem:0kB slab_reclaimable:52252kB slab_unreclaimable:40048kB kernel_stack:1304kB pagetables:1216kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9946*4kB 23*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 46080kB 69933 total pagecache pages 11 pages in swap cache Swap cache stats: add 37771, delete 37760, find 19255/25684 Free swap = 4120276kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70196 pages shared 392118 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1592] 498 1592 58372 1 0 0 0 munged [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1592 (munged) score 1 or sacrifice child Killed process 1592, UID 498, (munged) total-vm:233488kB, anon-rss:0kB, file-rss:4kB sendmail invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 sendmail cpuset=/ mems_allowed=0 Pid: 1639, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? getnstimeofday+0x60/0xf0 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 29 active_anon:0 inactive_anon:4 isolated_anon:0 active_file:208394 inactive_file:208245 isolated_file:608 unevictable:0 dirty:287 writeback:1 unstable:0 free:13336 slab_reclaimable:13152 slab_unreclaimable:10004 mapped:1 shmem:0 pagetables:280 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:45000kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:4kB active_file:829828kB inactive_file:829628kB unevictable:0kB isolated(anon):0kB isolated(file):2432kB present:2052064kB mlocked:0kB dirty:1148kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52364kB slab_unreclaimable:40000kB kernel_stack:1256kB pagetables:1120kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4029689 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9684*4kB 17*8kB 3*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 45000kB 70198 total pagecache pages 4 pages in swap cache Swap cache stats: add 37838, delete 37834, find 19294/25738 Free swap = 4121296kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70822 pages shared 391809 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1615] 0 1615 5523 1 0 0 0 xinetd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1615 (xinetd) score 1 or sacrifice child Killed process 1615, UID 0, (xinetd) total-vm:22092kB, anon-rss:0kB, file-rss:4kB Lustre: 700:0:(service.c:1889:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 9s req@ffff88007d98b000 x1436033302247472/t0(0) o400->d212be8a-cd9e-b654-7c84-9fe1868ce834@10.10.4.152@tcp:0/0 lens 192/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 Lustre: 700:0:(service.c:1889:ptlrpc_server_handle_req_in()) Skipped 1 previous similar message auditd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=-17, oom_score_adj=-1000 auditd cpuset=/ mems_allowed=0 Pid: 1131, comm: auditd Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __page_cache_alloc+0x87/0x90 [] ? find_get_page+0x1e/0xa0 [] ? filemap_fault+0x1a7/0x500 [] ? __do_fault+0x54/0x530 [] ? dequeue_entity+0x113/0x2e0 [] ? handle_pte_fault+0xf7/0xb50 [] ? thread_return+0x4e/0x76e [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? lock_hrtimer_base+0x31/0x60 [] ? hrtimer_try_to_cancel+0x3f/0xd0 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? ep_poll+0x306/0x330 [] ? default_wake_function+0x0/0x20 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 31 active_anon:0 inactive_anon:4 isolated_anon:0 active_file:225766 inactive_file:191230 isolated_file:288 unevictable:0 dirty:320 writeback:1 unstable:0 free:13153 slab_reclaimable:13188 slab_unreclaimable:10133 mapped:1 shmem:0 pagetables:266 bounce:0 Node 0 DMA free:8344kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:12kB active_file:3748kB inactive_file:3352kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:36kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:16kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10982 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:44268kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:4kB active_file:899316kB inactive_file:761568kB unevictable:0kB isolated(anon):0kB isolated(file):1152kB present:2052064kB mlocked:0kB dirty:1244kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52508kB slab_unreclaimable:40516kB kernel_stack:1248kB pagetables:1064kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3488652 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 50*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8344kB Node 0 DMA32: 9489*4kB 19*8kB 5*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44268kB 70219 total pagecache pages 4 pages in swap cache Swap cache stats: add 37939, delete 37935, find 19346/25811 Free swap = 4121532kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70527 pages shared 392237 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1623] 38 1623 7005 1 0 0 0 ntpd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1623 (ntpd) score 1 or sacrifice child Killed process 1623, UID 38, (ntpd) total-vm:28020kB, anon-rss:0kB, file-rss:4kB INFO: task mdt00_000:475 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt00_000 D 0000000000000000 0 475 2 0x00000080 ffff8800578fd820 0000000000000046 0000000000000002 0000000000000450 ffff8800578fd840 ffffffff811b566e ffff880061a30f00 ffff88006215bbc0 ffff88005c299af8 ffff8800578fdfd8 000000000000fb88 ffff88005c299af8 Call Trace: [] ? bh_lru_install+0x16e/0x1a0 [] ? prepare_to_wait+0x4e/0x80 [] do_get_write_access+0x29d/0x520 [jbd2] [] ? __find_get_block+0x97/0xe0 [] ? wake_bit_function+0x0/0x50 [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] [] __ldiskfs_journal_get_write_access+0x38/0x80 [ldiskfs] [] ldiskfs_reserve_inode_write+0x73/0xa0 [ldiskfs] [] ldiskfs_mark_inode_dirty+0x4c/0x1f0 [ldiskfs] [] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs] [] osd_attr_set+0x181/0x540 [osd_ldiskfs] [] lod_attr_set+0x12b/0x450 [lod] [] mdd_attr_set_internal+0x151/0x230 [mdd] [] mdd_attr_check_set_internal+0x275/0x2c0 [mdd] [] mdd_unlink+0x7a4/0xe30 [mdd] [] mdo_unlink+0x18/0x50 [mdt] [] mdt_reint_unlink+0x820/0x1010 [mdt] [] mdt_reint_rec+0x41/0xe0 [mdt] [] mdt_reint_internal+0x4c3/0x780 [mdt] [] mdt_reint+0x44/0xe0 [mdt] [] mdt_handle_common+0x648/0x1660 [mdt] [] mds_regular_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? ptlrpc_start_thread+0x2eb/0xe10 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 INFO: task mdt00_002:477 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt00_002 D 0000000000000000 0 477 2 0x00000080 ffff880051843820 0000000000000046 0000000000000002 0000000000000450 ffff880051843840 ffffffff811b566e ffff880061a30f00 ffff880067097f68 ffff88005c298638 ffff880051843fd8 000000000000fb88 ffff88005c298638 Call Trace: [] ? bh_lru_install+0x16e/0x1a0 [] ? prepare_to_wait+0x4e/0x80 [] do_get_write_access+0x29d/0x520 [jbd2] [] ? __find_get_block+0x97/0xe0 [] ? wake_bit_function+0x0/0x50 [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] [] __ldiskfs_journal_get_write_access+0x38/0x80 [ldiskfs] [] ldiskfs_reserve_inode_write+0x73/0xa0 [ldiskfs] [] ldiskfs_mark_inode_dirty+0x4c/0x1f0 [ldiskfs] [] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs] [] osd_attr_set+0x181/0x540 [osd_ldiskfs] [] lod_attr_set+0x12b/0x450 [lod] [] mdd_attr_set_internal+0x151/0x230 [mdd] [] mdd_attr_check_set_internal+0x275/0x2c0 [mdd] [] mdd_unlink+0x7a4/0xe30 [mdd] [] mdo_unlink+0x18/0x50 [mdt] [] mdt_reint_unlink+0x820/0x1010 [mdt] [] mdt_reint_rec+0x41/0xe0 [mdt] [] mdt_reint_internal+0x4c3/0x780 [mdt] [] mdt_reint+0x44/0xe0 [mdt] [] mdt_handle_common+0x648/0x1660 [mdt] [] mds_regular_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? ptlrpc_start_thread+0x2eb/0xe10 [ptlrpc] [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 sendmail invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 sendmail cpuset=/ mems_allowed=0 Pid: 1647, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? handle_mm_fault+0x23a/0x310 [] ? cpumask_any_but+0x31/0x50 [] ? __do_page_fault+0x139/0x480 [] ? handle_pte_fault+0x2cd/0xb50 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_unrolled+0x3e/0xb0 [] ? copy_user_generic+0xe/0x20 [] ? save_i387_xstate+0x1c7/0x220 [] ? do_signal+0x6f6/0x800 [] ? thread_return+0x4e/0x76e [] ? do_notify_resume+0x90/0xc0 [] ? int_signal+0x12/0x17 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:208201 inactive_file:207978 isolated_file:992 unevictable:0 dirty:215 writeback:1 unstable:0 free:12246 slab_reclaimable:13185 slab_unreclaimable:11234 mapped:1 shmem:0 pagetables:248 bounce:0 Node 0 DMA free:8352kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3652kB inactive_file:3448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:72kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:248kB slab_unreclaimable:24kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11387 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:40632kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:829152kB inactive_file:828464kB unevictable:0kB isolated(anon):0kB isolated(file):3968kB present:2052064kB mlocked:0kB dirty:788kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52492kB slab_unreclaimable:44912kB kernel_stack:1248kB pagetables:992kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:31835 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 52*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8352kB Node 0 DMA32: 8546*4kB 32*8kB 7*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 40632kB 70100 total pagecache pages 0 pages in swap cache Swap cache stats: add 37947, delete 37947, find 19348/25816 Free swap = 4121980kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 71115 pages shared 392696 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1639] 0 1639 22194 0 0 0 0 sendmail [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1639 (sendmail) score 1 or sacrifice child Killed process 1639, UID 0, (sendmail) total-vm:88776kB, anon-rss:0kB, file-rss:0kB INFO: task mdt00_000:475 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt00_000 D 0000000000000000 0 475 2 0x00000080 ffff8800578fd820 0000000000000046 0000000000000002 0000000000000450 ffff8800578fd840 ffffffff811b566e ffff880061a30f00 ffff88006215bbc0 ffff88005c299af8 ffff8800578fdfd8 000000000000fb88 ffff88005c299af8 Call Trace: [] ? bh_lru_install+0x16e/0x1a0 [] ? prepare_to_wait+0x4e/0x80 [] do_get_write_access+0x29d/0x520 [jbd2] [] ? __find_get_block+0x97/0xe0 [] ? wake_bit_function+0x0/0x50 [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] [] __ldiskfs_journal_get_write_access+0x38/0x80 [ldiskfs] [] ldiskfs_reserve_inode_write+0x73/0xa0 [ldiskfs] [] ldiskfs_mark_inode_dirty+0x4c/0x1f0 [ldiskfs] [] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs] [] osd_attr_set+0x181/0x540 [osd_ldiskfs] [] lod_attr_set+0x12b/0x450 [lod] [] mdd_attr_set_internal+0x151/0x230 [mdd] [] mdd_attr_check_set_internal+0x275/0x2c0 [mdd] [] mdd_unlink+0x7a4/0xe30 [mdd] [] mdo_unlink+0x18/0x50 [mdt] [] mdt_reint_unlink+0x820/0x1010 [mdt] [] mdt_reint_rec+0x41/0xe0 [mdt] [] mdt_reint_internal+0x4c3/0x780 [mdt] [] mdt_reint+0x44/0xe0 [mdt] [] mdt_handle_common+0x648/0x1660 [mdt] [] mds_regular_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? ptlrpc_start_thread+0x2eb/0xe10 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 INFO: task mdt00_002:477 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt00_002 D 0000000000000000 0 477 2 0x00000080 ffff880051843820 0000000000000046 0000000000000002 0000000000000450 ffff880051843840 ffffffff811b566e ffff880061a30f00 ffff880067097f68 ffff88005c298638 ffff880051843fd8 000000000000fb88 ffff88005c298638 Call Trace: [] ? bh_lru_install+0x16e/0x1a0 [] ? prepare_to_wait+0x4e/0x80 [] do_get_write_access+0x29d/0x520 [jbd2] [] ? __find_get_block+0x97/0xe0 [] ? wake_bit_function+0x0/0x50 [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] [] __ldiskfs_journal_get_write_access+0x38/0x80 [ldiskfs] [] ldiskfs_reserve_inode_write+0x73/0xa0 [ldiskfs] [] ldiskfs_mark_inode_dirty+0x4c/0x1f0 [ldiskfs] [] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs] [] osd_attr_set+0x181/0x540 [osd_ldiskfs] [] lod_attr_set+0x12b/0x450 [lod] [] mdd_attr_set_internal+0x151/0x230 [mdd] [] mdd_attr_check_set_internal+0x275/0x2c0 [mdd] [] mdd_unlink+0x7a4/0xe30 [mdd] [] mdo_unlink+0x18/0x50 [mdt] [] mdt_reint_unlink+0x820/0x1010 [mdt] [] mdt_reint_rec+0x41/0xe0 [mdt] [] mdt_reint_internal+0x4c3/0x780 [mdt] [] mdt_reint+0x44/0xe0 [mdt] [] mdt_handle_common+0x648/0x1660 [mdt] [] mds_regular_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? ptlrpc_start_thread+0x2eb/0xe10 [ptlrpc] [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 auditd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=-17, oom_score_adj=-1000 auditd cpuset=/ mems_allowed=0 Pid: 1131, comm: auditd Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __page_cache_alloc+0x87/0x90 [] ? find_get_page+0x1e/0xa0 [] ? filemap_fault+0x1a7/0x500 [] ? __do_fault+0x54/0x530 [] ? dequeue_entity+0x113/0x2e0 [] ? handle_pte_fault+0xf7/0xb50 [] ? thread_return+0x4e/0x76e [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? lock_hrtimer_base+0x31/0x60 [] ? hrtimer_try_to_cancel+0x3f/0xd0 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? ep_poll+0x306/0x330 [] ? default_wake_function+0x0/0x20 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 30 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:208046 inactive_file:207997 isolated_file:960 unevictable:0 dirty:35 writeback:1 unstable:0 free:12409 slab_reclaimable:13184 slab_unreclaimable:11256 mapped:1 shmem:0 pagetables:205 bounce:0 Node 0 DMA free:8352kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3652kB inactive_file:3448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:36kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:248kB slab_unreclaimable:24kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11387 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:41284kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:828532kB inactive_file:828540kB unevictable:0kB isolated(anon):0kB isolated(file):3840kB present:2052064kB mlocked:0kB dirty:104kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52488kB slab_unreclaimable:45000kB kernel_stack:1248kB pagetables:820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2630125 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 52*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8352kB Node 0 DMA32: 8647*4kB 47*8kB 13*16kB 3*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 41284kB 69935 total pagecache pages 0 pages in swap cache Swap cache stats: add 37950, delete 37950, find 19349/25818 Free swap = 4123652kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70914 pages shared 392717 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1647] 51 1647 19539 0 0 0 0 sendmail [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1647 (sendmail) score 1 or sacrifice child Killed process 1647, UID 51, (sendmail) total-vm:78156kB, anon-rss:0kB, file-rss:0kB sendmail: page allocation failure. order:0, mode:0x200da Pid: 1647, comm: sendmail Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? __alloc_pages_nodemask+0x757/0x8d0 [] ? alloc_pages_vma+0x9a/0x150 [] ? read_swap_cache_async+0xf2/0x150 [] ? valid_swaphandles+0x69/0x150 [] ? swapin_readahead+0x87/0xc0 [] ? handle_pte_fault+0x70b/0xb50 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? handle_mm_fault+0x23a/0x310 [] ? cpumask_any_but+0x31/0x50 [] ? __do_page_fault+0x139/0x480 [] ? handle_pte_fault+0x2cd/0xb50 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? copy_user_generic_unrolled+0x3e/0xb0 [] ? copy_user_generic+0xe/0x20 [] ? save_i387_xstate+0x1c7/0x220 [] ? do_signal+0x6f6/0x800 [] ? thread_return+0x4e/0x76e [] ? do_notify_resume+0x90/0xc0 [] ? int_signal+0x12/0x17 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:208135 inactive_file:207928 isolated_file:928 unevictable:0 dirty:33 writeback:1 unstable:0 free:12440 slab_reclaimable:13184 slab_unreclaimable:11256 mapped:1 shmem:0 pagetables:205 bounce:0 Node 0 DMA free:8352kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3652kB inactive_file:3448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:36kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:248kB slab_unreclaimable:24kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11387 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:41408kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:828888kB inactive_file:828264kB unevictable:0kB isolated(anon):0kB isolated(file):3712kB present:2052064kB mlocked:0kB dirty:96kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52488kB slab_unreclaimable:45000kB kernel_stack:1248kB pagetables:820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 52*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8352kB Node 0 DMA32: 8678*4kB 47*8kB 13*16kB 3*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 41408kB 69933 total pagecache pages 0 pages in swap cache Swap cache stats: add 37950, delete 37950, find 19349/25818 Free swap = 4123652kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70878 pages shared 392720 pages non-shared auditd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=-17, oom_score_adj=-1000 auditd cpuset=/ mems_allowed=0 Pid: 1131, comm: auditd Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __page_cache_alloc+0x87/0x90 [] ? find_get_page+0x1e/0xa0 [] ? filemap_fault+0x1a7/0x500 [] ? __do_fault+0x54/0x530 [] ? dequeue_entity+0x113/0x2e0 [] ? handle_pte_fault+0xf7/0xb50 [] ? thread_return+0x4e/0x76e [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? lock_hrtimer_base+0x31/0x60 [] ? hrtimer_try_to_cancel+0x3f/0xd0 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? ep_poll+0x306/0x330 [] ? default_wake_function+0x0/0x20 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 34 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:208036 inactive_file:208024 isolated_file:928 unevictable:0 dirty:13 writeback:1 unstable:0 free:12488 slab_reclaimable:13151 slab_unreclaimable:11258 mapped:1 shmem:0 pagetables:167 bounce:0 Node 0 DMA free:8352kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3652kB inactive_file:3448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:36kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:248kB slab_unreclaimable:24kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11387 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:41600kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:828492kB inactive_file:828648kB unevictable:0kB isolated(anon):0kB isolated(file):3712kB present:2052064kB mlocked:0kB dirty:16kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52356kB slab_unreclaimable:45008kB kernel_stack:1240kB pagetables:668kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4955655 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 52*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8352kB Node 0 DMA32: 8680*4kB 56*8kB 18*16kB 4*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 41600kB 69920 total pagecache pages 0 pages in swap cache Swap cache stats: add 37952, delete 37952, find 19351/25821 Free swap = 4125052kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70862 pages shared 392651 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1669] 0 1669 29303 1 0 0 0 crond [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1669 (crond) score 1 or sacrifice child Killed process 1669, UID 0, (crond) total-vm:117212kB, anon-rss:0kB, file-rss:4kB ll_mgs_0000 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 ll_mgs_0000 cpuset=/ mems_allowed=0 Pid: 469, comm: ll_mgs_0000 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:208063 inactive_file:208061 isolated_file:864 unevictable:0 dirty:13 writeback:1 unstable:0 free:12402 slab_reclaimable:13140 slab_unreclaimable:11408 mapped:1 shmem:0 pagetables:153 bounce:0 Node 0 DMA free:8352kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3652kB inactive_file:3448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:36kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:248kB slab_unreclaimable:24kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11387 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:41256kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:828600kB inactive_file:828796kB unevictable:0kB isolated(anon):0kB isolated(file):3456kB present:2052064kB mlocked:0kB dirty:16kB writeback:4kB mapped:4kB shmem:0kB slab_reclaimable:52312kB slab_unreclaimable:45608kB kernel_stack:1240kB pagetables:612kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 52*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8352kB Node 0 DMA32: 8590*4kB 58*8kB 18*16kB 4*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 41256kB 69920 total pagecache pages 0 pages in swap cache Swap cache stats: add 37952, delete 37952, find 19351/25821 Free swap = 4125656kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70797 pages shared 392835 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1680] 0 1680 5363 0 0 0 0 atd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1680 (atd) score 1 or sacrifice child Killed process 1680, UID 0, (atd) total-vm:21452kB, anon-rss:0kB, file-rss:0kB Lustre: 700:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (18:296s); client may timeout. req@ffff880030db1400 x1436032519867016/t0(0) o400->lustre-MDT0000-lwp-OST0005_UUID@10.10.4.151@tcp:0/0 lens 224/192 e 0 to 0 dl 1369550075 ref 1 fl Complete:H/0/0 rc 0/0 Lustre: 700:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 26 previous similar messages LNet: Service thread pid 700 completed after 314.51s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: mdt: This server is not able to keep up with request traffic (cpu-bound). Lustre: 700:0:(service.c:1500:ptlrpc_at_check_timed()) earlyQ=6 reqQ=0 recA=3, svcEst=314, delay=310064(jiff) Lustre: 700:0:(service.c:1301:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-295s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff88002174ac00 x1436074024334308/t0(0) o36->4cdcd8f7-da74-e70d-a566-17cc3f711c5f@10.10.4.153@tcp:0/0 lens 456/696 e 0 to 0 dl 1369550076 ref 2 fl Interpret:/0/0 rc 0/0 Lustre: 700:0:(service.c:1301:ptlrpc_at_send_early_reply()) Skipped 23 previous similar messages auditd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=-17, oom_score_adj=-1000 auditd cpuset=/ mems_allowed=0 Pid: 1131, comm: auditd Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __page_cache_alloc+0x87/0x90 [] ? find_get_page+0x1e/0xa0 [] ? filemap_fault+0x1a7/0x500 [] ? __do_fault+0x54/0x530 [] ? dequeue_entity+0x113/0x2e0 [] ? handle_pte_fault+0xf7/0xb50 [] ? thread_return+0x4e/0x76e [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? lock_hrtimer_base+0x31/0x60 [] ? hrtimer_try_to_cancel+0x3f/0xd0 [] ? handle_mm_fault+0x23a/0x310 [] ? __do_page_fault+0x139/0x480 [] ? ep_poll+0x306/0x330 [] ? default_wake_function+0x0/0x20 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:208123 inactive_file:208115 isolated_file:736 unevictable:0 dirty:0 writeback:0 unstable:0 free:13775 slab_reclaimable:13140 slab_unreclaimable:10067 mapped:1 shmem:0 pagetables:139 bounce:0 Node 0 DMA free:8352kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3652kB inactive_file:3448kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:248kB slab_unreclaimable:24kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11387 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:46748kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:828840kB inactive_file:829012kB unevictable:0kB isolated(anon):0kB isolated(file):2944kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:52312kB slab_unreclaimable:40244kB kernel_stack:1240kB pagetables:556kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2429986 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 52*4kB 32*8kB 25*16kB 20*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8352kB Node 0 DMA32: 9921*4kB 67*8kB 18*16kB 5*32kB 3*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 46748kB 69906 total pagecache pages 0 pages in swap cache Swap cache stats: add 37952, delete 37952, find 19351/25821 Free swap = 4125836kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 70651 pages shared 391552 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1705] 0 1705 1018 1 0 0 0 agetty [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1705 (agetty) score 1 or sacrifice child Killed process 1705, UID 0, (agetty) total-vm:4072kB, anon-rss:0kB, file-rss:4kB LustreError: 700:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.10.4.151@tcp: deadline 18:296s ago req@ffff8800378ffc00 x1436032519867004/t0(0) o400->lustre-MDT0000-lwp-OST0002_UUID@10.10.4.151@tcp:0/0 lens 224/0 e 0 to 0 dl 1369550075 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 LustreError: 700:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 21 previous similar messages init: serial (ttyS0) main process (1705) killed by KILL signal init: serial (ttyS0) main process ended, respawning LNet: Service thread pid 475 completed after 353.48s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LNet: Skipped 1 previous similar message Lustre: lustre-MDT0000: Client 4cdcd8f7-da74-e70d-a566-17cc3f711c5f (at 10.10.4.153@tcp) reconnecting Lustre: Skipped 1 previous similar message LNetError: 12885:0:(socklnd.c:1659:ksocknal_destroy_conn()) Completing partial receive from 12345-10.10.4.152@tcp[1], ip 10.10.4.152:1023, with error, wanted: 192, left: 192, last alive is 315 secs ago LustreError: 12885:0:(events.c:306:request_in_callback()) event type 2, status -5, service mdt LustreError: 475:0:(pack_generic.c:593:__lustre_unpack_msg()) message length 0 too small for magic/version check LustreError: 475:0:(sec.c:2057:sptlrpc_svc_unwrap_request()) error unpacking request from 12345-10.10.4.152@tcp x1436033302247505 LustreError: 11-0: lustre-OST0000-osc-MDT0000: Communicating with 10.10.4.151@tcp, operation obd_ping failed with -107. LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.10.4.151@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: 167-0: lustre-OST0004-osc-MDT0000: This client was evicted by lustre-OST0004; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to lustre-OST0000 (at 10.10.4.151@tcp) Lustre: Skipped 1 previous similar message LustreError: 11-0: lustre-OST0003-osc-MDT0000: Communicating with 10.10.4.151@tcp, operation obd_ping failed with -107. LustreError: Skipped 4 previous similar messages Lustre: 12891:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1369550047/real 1369550047] req@ffff880037b93c00 x1436032537207612/t0(0) o400->MGC10.10.4.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1369550098 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 166-1: MGC10.10.4.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LNet: Service thread pid 1114 completed after 639.44s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LNet: Skipped 1 previous similar message Lustre: 12889:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1369550376/real 0] req@ffff88001d3ed000 x1436032537207868/t0(0) o250->MGC10.10.4.150@tcp@0@lo:26/25 lens 400/544 e 0 to 1 dl 1369550382 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 12889:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Lustre: lustre-MDT0000: haven't heard from client d212be8a-cd9e-b654-7c84-9fe1868ce834 (at 10.10.4.152@tcp) in 314 seconds. I think it's dead, and I am evicting it. exp ffff88002162f400, cur 1369550388 expire 1369550358 last 1369550074 LNetError: 12881:0:(socklnd_cb.c:1731:ksocknal_recv_hello()) Error -104 reading HELLO from 10.10.4.152 LNetError: 11b-b: Connection to 10.10.4.152@tcp at host 10.10.4.152 on port 7988 was reset: is it running a compatible version of Lustre and is 10.10.4.152@tcp one of its NIDs? LNet: Service thread pid 471 completed after 395.55s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 12889:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1369550386/real 0] req@ffff88002809f800 x1436032537207920/t0(0) o250->MGC10.10.4.150@tcp@0@lo:26/25 lens 400/544 e 0 to 1 dl 1369550397 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 ll_mgs_0000 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 ll_mgs_0000 cpuset=/ mems_allowed=0 Pid: 469, comm: ll_mgs_0000 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:207281 inactive_file:206914 isolated_file:128 unevictable:0 dirty:0 writeback:0 unstable:0 free:17426 slab_reclaimable:12954 slab_unreclaimable:9215 mapped:1 shmem:0 pagetables:132 bounce:0 Node 0 DMA free:8356kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3616kB inactive_file:3368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:108kB kernel_stack:40kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11871 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:61348kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:825508kB inactive_file:824288kB unevictable:0kB isolated(anon):0kB isolated(file):512kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:51572kB slab_unreclaimable:36752kB kernel_stack:1192kB pagetables:528kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1542502 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 55*4kB 33*8kB 26*16kB 19*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8356kB Node 0 DMA32: 13013*4kB 204*8kB 59*16kB 18*32kB 4*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 61348kB 67262 total pagecache pages 0 pages in swap cache Swap cache stats: add 38078, delete 38078, find 19412/25905 Free swap = 4125924kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 67398 pages shared 391061 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1707] 0 1707 1015 1 0 0 0 mingetty [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1707 (mingetty) score 1 or sacrifice child Killed process 1707, UID 0, (mingetty) total-vm:4060kB, anon-rss:0kB, file-rss:4kB init invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0 init cpuset=/ mems_allowed=0 Pid: 1, comm: init Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? alloc_fd+0x92/0x160 [] ? fd_install+0x47/0x90 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:207310 inactive_file:206914 isolated_file:96 unevictable:0 dirty:0 writeback:0 unstable:0 free:17468 slab_reclaimable:12934 slab_unreclaimable:9207 mapped:1 shmem:0 pagetables:125 bounce:0 Node 0 DMA free:8356kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3616kB inactive_file:3368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:108kB kernel_stack:40kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11871 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:61516kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:825624kB inactive_file:824288kB unevictable:0kB isolated(anon):0kB isolated(file):384kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:51492kB slab_unreclaimable:36720kB kernel_stack:1192kB pagetables:500kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 55*4kB 33*8kB 26*16kB 19*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8356kB Node 0 DMA32: 13039*4kB 208*8kB 61*16kB 18*32kB 4*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 61516kB 67262 total pagecache pages 0 pages in swap cache Swap cache stats: add 38078, delete 38078, find 19412/25905 Free swap = 4126008kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 67365 pages shared 391060 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1709] 0 1709 1015 1 0 0 0 mingetty [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1709 (mingetty) score 1 or sacrifice child Killed process 1709, UID 0, (mingetty) total-vm:4060kB, anon-rss:0kB, file-rss:4kB ll_mgs_0000 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 ll_mgs_0000 cpuset=/ mems_allowed=0 Pid: 469, comm: ll_mgs_0000 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:207270 inactive_file:206932 isolated_file:128 unevictable:0 dirty:0 writeback:0 unstable:0 free:17482 slab_reclaimable:12934 slab_unreclaimable:9207 mapped:1 shmem:0 pagetables:117 bounce:0 Node 0 DMA free:8356kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3616kB inactive_file:3368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:108kB kernel_stack:40kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11871 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:61572kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:825464kB inactive_file:824360kB unevictable:0kB isolated(anon):0kB isolated(file):512kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:51492kB slab_unreclaimable:36720kB kernel_stack:1192kB pagetables:468kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2187440 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 55*4kB 33*8kB 26*16kB 19*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8356kB Node 0 DMA32: 13039*4kB 211*8kB 63*16kB 18*32kB 4*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 61572kB 67262 total pagecache pages 0 pages in swap cache Swap cache stats: add 38078, delete 38078, find 19412/25905 Free swap = 4126088kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 67396 pages shared 391013 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1711] 0 1711 1015 1 0 0 0 mingetty [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1711 (mingetty) score 1 or sacrifice child Killed process 1711, UID 0, (mingetty) total-vm:4060kB, anon-rss:0kB, file-rss:4kB ll_mgs_0000 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 ll_mgs_0000 cpuset=/ mems_allowed=0 Pid: 469, comm: ll_mgs_0000 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:207197 inactive_file:207037 isolated_file:96 unevictable:0 dirty:0 writeback:0 unstable:0 free:17515 slab_reclaimable:12928 slab_unreclaimable:9195 mapped:1 shmem:0 pagetables:109 bounce:0 Node 0 DMA free:8356kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3616kB inactive_file:3368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:108kB kernel_stack:40kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11871 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:61704kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:825172kB inactive_file:824780kB unevictable:0kB isolated(anon):0kB isolated(file):384kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:51468kB slab_unreclaimable:36672kB kernel_stack:1192kB pagetables:436kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:612711 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 55*4kB 33*8kB 26*16kB 19*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8356kB Node 0 DMA32: 13052*4kB 213*8kB 67*16kB 18*32kB 4*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 61704kB 67262 total pagecache pages 0 pages in swap cache Swap cache stats: add 38078, delete 38078, find 19412/25905 Free swap = 4126172kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 67363 pages shared 391014 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1713] 0 1713 1015 1 0 0 0 mingetty [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1713 (mingetty) score 1 or sacrifice child Killed process 1713, UID 0, (mingetty) total-vm:4060kB, anon-rss:0kB, file-rss:4kB init invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0 init cpuset=/ mems_allowed=0 Pid: 1, comm: init Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? __delayacct_freepages_end+0x2e/0x30 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? alloc_fd+0x92/0x160 [] ? fd_install+0x47/0x90 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:207239 inactive_file:206963 isolated_file:128 unevictable:0 dirty:0 writeback:0 unstable:0 free:17530 slab_reclaimable:12928 slab_unreclaimable:9194 mapped:1 shmem:0 pagetables:101 bounce:0 Node 0 DMA free:8356kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3616kB inactive_file:3368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:108kB kernel_stack:40kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11871 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:61764kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:825340kB inactive_file:824484kB unevictable:0kB isolated(anon):0kB isolated(file):512kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:51468kB slab_unreclaimable:36668kB kernel_stack:1192kB pagetables:404kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2267032 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 55*4kB 33*8kB 26*16kB 19*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8356kB Node 0 DMA32: 13057*4kB 218*8kB 67*16kB 18*32kB 4*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 61764kB 67262 total pagecache pages 0 pages in swap cache Swap cache stats: add 38078, delete 38078, find 19412/25905 Free swap = 4126252kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 67394 pages shared 390967 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1716] 0 1716 1015 1 0 0 0 mingetty [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1716 (mingetty) score 1 or sacrifice child Killed process 1716, UID 0, (mingetty) total-vm:4060kB, anon-rss:0kB, file-rss:4kB ll_mgs_0000 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=0 ll_mgs_0000 cpuset=/ mems_allowed=0 Pid: 469, comm: ll_mgs_0000 Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? security_real_capable_noaudit+0x3c/0x70 [] ? oom_kill_process+0x82/0x2a0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x220/0x3c0 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? do_fork+0x94/0x460 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? kernel_thread+0x82/0xe0 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 [] ? cfs_create_thread+0x65/0xa0 [libcfs] [] ? ptlrpc_start_thread+0x7aa/0xe10 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0x858/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:207339 inactive_file:206899 isolated_file:96 unevictable:0 dirty:0 writeback:0 unstable:0 free:17544 slab_reclaimable:12928 slab_unreclaimable:9194 mapped:1 shmem:0 pagetables:101 bounce:0 Node 0 DMA free:8356kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3616kB inactive_file:3368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:108kB kernel_stack:40kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11871 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:61820kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:825740kB inactive_file:824228kB unevictable:0kB isolated(anon):0kB isolated(file):384kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:51468kB slab_unreclaimable:36668kB kernel_stack:1192kB pagetables:404kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 55*4kB 33*8kB 26*16kB 19*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8356kB Node 0 DMA32: 13071*4kB 218*8kB 67*16kB 18*32kB 4*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 61820kB 67262 total pagecache pages 0 pages in swap cache Swap cache stats: add 38078, delete 38078, find 19412/25905 Free swap = 4126328kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 67361 pages shared 390977 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1714] 0 1714 2725 0 0 -17 -1000 udevd [ 1718] 0 1718 1015 1 0 0 0 mingetty Out of memory: Kill process 1718 (mingetty) score 1 or sacrifice child Killed process 1718, UID 0, (mingetty) total-vm:4060kB, anon-rss:0kB, file-rss:4kB init invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0 init cpuset=/ mems_allowed=0 Pid: 1, comm: init Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 [] ? dump_header+0x90/0x1b0 [] ? select_bad_process+0x9e/0x120 [] ? out_of_memory+0x37a/0x3c0 [] ? drain_local_pages+0x0/0x20 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? alloc_fd+0x92/0x160 [] ? fd_install+0x47/0x90 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:207156 inactive_file:207049 isolated_file:128 unevictable:0 dirty:0 writeback:0 unstable:0 free:17581 slab_reclaimable:12925 slab_unreclaimable:9171 mapped:1 shmem:0 pagetables:85 bounce:0 Node 0 DMA free:8360kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:3456kB inactive_file:3528kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:244kB slab_unreclaimable:104kB kernel_stack:40kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:6440 all_unreclaimable? no lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:61964kB min:44720kB low:55900kB high:67080kB active_anon:0kB inactive_anon:0kB active_file:825168kB inactive_file:824668kB unevictable:0kB isolated(anon):0kB isolated(file):512kB present:2052064kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:51456kB slab_unreclaimable:36580kB kernel_stack:1192kB pagetables:340kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1145712 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 54*4kB 34*8kB 26*16kB 19*32kB 15*64kB 6*128kB 2*256kB 1*512kB 2*1024kB 1*2048kB 0*4096kB = 8360kB Node 0 DMA32: 13083*4kB 222*8kB 69*16kB 19*32kB 4*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 61964kB 67265 total pagecache pages 0 pages in swap cache Swap cache stats: add 38078, delete 38078, find 19412/25905 Free swap = 4126412kB Total swap = 4128760kB 524284 pages RAM 43668 pages reserved 67395 pages shared 390917 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 472] 0 472 2726 0 0 -17 -1000 udevd [ 787] 0 787 2660 0 0 -17 -1000 udevd [ 1131] 0 1131 23299 1 0 -17 -1000 auditd [ 1607] 0 1607 16028 0 0 -17 -1000 sshd [ 1714] 0 1714 2725 0 0 -17 -1000 udevd Kernel panic - not syncing: Out of memory and no killable processes... Pid: 1, comm: init Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? panic+0xa7/0x16f [] ? dump_header+0x101/0x1b0 [] ? out_of_memory+0x38f/0x3c0 [] ? drain_local_pages+0x0/0x20 [] ? __alloc_pages_nodemask+0x8ac/0x8d0 [] ? alloc_pages_current+0xaa/0x110 [] ? __get_free_pages+0xe/0x50 [] ? copy_process+0xe4/0x1450 [] ? __do_page_fault+0x1ec/0x480 [] ? do_fork+0x94/0x460 [] ? alloc_fd+0x92/0x160 [] ? fd_install+0x47/0x90 [] ? sys_clone+0x28/0x30 [] ? stub_clone+0x13/0x20 [] ? system_call_fastpath+0x16/0x1b Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 (jenkins@builder-4-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Thu May 23 10:02:55 PDT 2013 Command line: ro root=UUID=08cf7b2b-0dec-4c87-87dd-e442d5c4f95f rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off memmap=exactmap memmap=613K@4K memmap=131463K@49765K elfcorehdr=181228K memmap=4K$0K memmap=64K$960K memmap=12K$2097140K memmap=272K$4194032K KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls Disabled fast string operations BIOS-provided physical RAM map: BIOS-e820: 0000000000000100 - 000000000009a400 (usable) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 000000007fec3000 (usable) BIOS-e820: 000000007ff00000 - 000000007fffd000 (usable) BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) last_pfn = 0x7fffd max_arch_pfn = 0x400000000 user-defined physical RAM map: user: 0000000000000000 - 0000000000001000 (reserved) user: 0000000000001000 - 000000000009a400 (usable) user: 00000000000f0000 - 0000000000100000 (reserved) user: 0000000003099400 - 000000000b0fb000 (usable) user: 000000007fffd000 - 0000000080000000 (reserved) user: 00000000fffbc000 - 0000000100000000 (reserved) DMI 2.4 present. SMBIOS version 2.4 @ 0xFDA50 last_pfn = 0xb0fb max_arch_pfn = 0x400000000 PAT not supported by CPU. init_memory_mapping: 0000000000000000-000000000b0fb000 RAMDISK: 0acae000 - 0b0eea8a ACPI: RSDP 00000000000fda20 00014 (v00 BOCHS ) ACPI: RSDT 000000007fffd740 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) ACPI: FACP 000000007ffffe40 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) ACPI: DSDT 000000007fffd900 024FD (v01 BXPC BXDSDT 00000001 INTL 20090123) ACPI: FACS 000000007ffffe00 00040 ACPI: SSDT 000000007fffd860 0009E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) ACPI: APIC 000000007fffd770 00078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) Setting APIC routing to flat. No NUMA configuration found Faking a node at 0000000000000000-000000000b0fb000 Bootmem setup node 0 0000000000000000-000000000b0fb000 NODE_DATA [0000000000009000 - 000000000003cfff] bootmap [000000000003d000 - 000000000003e61f] pages 2 (7 early reservations) ==> bootmem [0000000000 - 000b0fb000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] #2 [0004000000 - 000501b0a4] TEXT DATA BSS ==> [0004000000 - 000501b0a4] #3 [000acae000 - 000b0eea8a] RAMDISK ==> [000acae000 - 000b0eea8a] #4 [000009a400 - 0000100000] BIOS reserved ==> [000009a400 - 0000100000] #5 [000501c000 - 000501c081] BRK ==> [000501c000 - 000501c081] #6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000] found SMP MP-table at [ffff8800000fda70] fda70 kvm-clock: Using msrs 4b564d01 and 4b564d00 kvm-clock: cpu 0, msr 0:4c256c1, boot clock Zone PFN ranges: DMA 0x00000001 -> 0x00001000 DMA32 0x00001000 -> 0x00100000 Normal 0x00100000 -> 0x00100000 Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000001 -> 0x0000009a 0: 0x0000309a -> 0x0000b0fb ACPI: PM-Timer IO Port: 0xb008 Setting APIC routing to flat. ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) Using ACPI (MADT) for SMP configuration information SMP: Allowing 1 CPUs, 0 hotplug CPUs PM: Registered nosave memory: 000000000009a000 - 00000000000f0000 PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 PM: Registered nosave memory: 0000000000100000 - 000000000309a000 Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) Booting paravirtualized kernel on KVM NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 PERCPU: Embedded 31 pages/cpu @ffff880003200000 s94616 r8192 d24168 u2097152 pcpu-alloc: s94616 r8192 d24168 u2097152 alloc=1*2097152 pcpu-alloc: [0] 0 kvm-clock: cpu 0, msr 0:32166c1, primary cpu clock kvm-stealtime: cpu 0, msr 320e840 Built 1 zonelists in Node order, mobility grouping on. Total pages: 32398 Policy zone: DMA32 Kernel command line: ro root=UUID=08cf7b2b-0dec-4c87-87dd-e442d5c4f95f rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off memmap=exactmap memmap=613K@4K memmap=131463K@49765K elfcorehdr=181228K memmap=4K$0K memmap=64K$960K memmap=12K$2097140K memmap=272K$4194032K Misrouted IRQ fixup and polling support enabled This may significantly impact system performance Disabling memory control group subsystem PID hash table entries: 512 (order: 0, 4096 bytes) Checking aperture... No AGP bridge found Memory: 106524k/181228k available (5222k kernel code, 49156k absent, 25548k reserved, 7119k data, 1264k init) Hierarchical RCU implementation. NR_IRQS:33024 nr_irqs:256 Spurious LAPIC timer interrupt on cpu 0 Console: colour VGA+ 80x25 console [ttyS0] enabled Detected 2400.084 MHz processor. Calibrating delay loop (skipped) preset value.. 4800.16 BogoMIPS (lpj=2400084) pid_max: default: 32768 minimum: 301 Security Framework initialized SELinux: Initializing. Dentry cache hash table entries: 16384 (order: 5, 131072 bytes) Inode-cache hash table entries: 8192 (order: 4, 65536 bytes) Mount-cache hash table entries: 256 Initializing cgroup subsys ns Initializing cgroup subsys cpuacct Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls Initializing cgroup subsys blkio Initializing cgroup subsys perf_event Initializing cgroup subsys net_prio Disabled fast string operations alternatives: switching to unfair spinlock SMP alternatives: switching to UP code Freeing SMP alternatives: 35k freed ACPI: Core revision 20090903 ftrace: converting mcount calls to 0f 1f 44 00 00 ftrace: allocating 21438 entries in 85 pages Setting APIC routing to flat ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 CPU0: Intel QEMU Virtual CPU version (cpu64-rhel6) stepping 03 Performance Events: Broken PMU hardware detected, using software events only. NMI watchdog disabled (cpu0): hardware events not enabled Brought up 1 CPUs Total of 1 processors activated (4800.16 BogoMIPS). devtmpfs: initialized regulator: core version 0.5 NET: Registered protocol family 16 ACPI: bus type pci registered PCI: Using configuration type 1 for base access bio: create slab at 0 ACPI: Interpreter enabled ACPI: (supports S0 S5) ACPI: Using IOAPIC for interrupt routing ACPI: No dock devices found. HEST: Table not found. PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) ACPI: PCI Interrupt Link [LNKB] (IRQs 5 10 11) *0, disabled. ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0 vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none vgaarb: loaded vgaarb: bridge control possible 0000:00:02.0 SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb PCI: Using ACPI for IRQ routing NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default Switching to clocksource kvm-clock pnp: PnP ACPI init ACPI: bus type pnp registered pnp: PnP ACPI: found 6 devices ACPI: ACPI bus type pnp unregistered NET: Registered protocol family 2 IP route cache hash table entries: 1024 (order: 1, 8192 bytes) TCP established hash table entries: 4096 (order: 4, 65536 bytes) TCP bind hash table entries: 4096 (order: 4, 65536 bytes) TCP: Hash tables configured (established 4096 bind 4096) TCP reno registered NET: Registered protocol family 1 pci 0000:00:00.0: Limiting direct PCI/PCI transfers pci 0000:00:01.0: Activating ISA DMA hang workarounds ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 pci 0000:00:01.2: PCI INT D disabled Trying to unpack rootfs image as initramfs... Freeing initrd memory: 4354k freed audit: initializing netlink socket (disabled) type=2000 audit(1369550406.230:1): initialized HugeTLB registered 2 MB page size, pre-allocated 0 pages VFS: Disk quotas dquot_6.5.2 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) msgmni has been set to 216 alg: No test for stdrng (krng) ksign: Installing public key data Loading keyring - Added public key 30BD1F01F604D494 - User ID: Red Hat, Inc. (Kernel Module GPG key) - Added public key D4A26C9CCD09BEDA - User ID: Red Hat Enterprise Linux Driver Update Program Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered (default) io scheduler cfq registered pci_hotplug: PCI Hot Plug PCI Core version: 0.5 pciehp: PCI Express Hot Plug Controller Driver version: 0.4 acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 acpiphp: Slot [1] registered acpiphp: Slot [2] registered acpiphp: Slot [3] registered acpiphp: Slot [4] registered acpiphp: Slot [5] registered acpiphp: Slot [6] registered acpiphp: Slot [7] registered acpiphp: Slot [8] registered acpiphp: Slot [9] registered acpiphp: Slot [10] registered acpiphp: Slot [11] registered acpiphp: Slot [12] registered acpiphp: Slot [13] registered acpiphp: Slot [14] registered acpiphp: Slot [15] registered acpiphp: Slot [16] registered acpiphp: Slot [17] registered acpiphp: Slot [18] registered acpiphp: Slot [19] registered acpiphp: Slot [20] registered acpiphp: Slot [21] registered acpiphp: Slot [22] registered acpiphp: Slot [23] registered acpiphp: Slot [24] registered acpiphp: Slot [25] registered acpiphp: Slot [26] registered acpiphp: Slot [27] registered acpiphp: Slot [28] registered acpiphp: Slot [29] registered acpiphp: Slot [30] registered acpiphp: Slot [31] registered ipmi message handler version 39.2 IPMI System Interface driver. ipmi_si: Adding default-specified kcs state machine ipmi_si: Trying default-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Adding default-specified smic state machine ipmi_si: Trying default-specified smic state machine at i/o address 0xca9, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Adding default-specified bt state machine ipmi_si: Trying default-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Unable to find any System Interface(s) input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 ACPI: Power Button [PWRF] ERST: Table is not found! GHES: HEST is not enabled! Non-volatile memory driver v1.3 Linux agpgart interface v0.103 crash memory driver: version 1.1 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled erial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A brd: module loaded loop: module loaded input: Macintosh mouse button emulation as /devices/virtual/input/input1 Fixed MDIO Bus: probed ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver uhci_hcd: USB Universal Host Controller Interface driver uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 uhci_hcd 0000:00:01.2: UHCI Host Controller uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 irq 11: nobody cared (try booting with the "irqpoll" option) Pid: 1, comm: swapper Not tainted 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 Call Trace: [] ? __report_bad_irq+0x2b/0xa0 [] ? note_interrupt+0x18c/0x1d0 [] ? handle_fasteoi_irq+0xcd/0xf0 [] ? handle_irq+0x49/0xa0 [] ? do_IRQ+0x6c/0xf0 [] ? ret_from_intr+0x0/0x11 [] ? __do_softirq+0x73/0x1e0 [] ? call_softirq+0x1c/0x30 [] ? do_softirq+0x65/0xa0 [] ? irq_exit+0x85/0x90 [] ? do_IRQ+0x75/0xf0 [] ? ret_from_intr+0x0/0x11 [] ? _spin_unlock_irqrestore+0x17/0x20 [] ? __setup_irq+0x1b3/0x3c0 [] ? usb_hcd_irq+0x0/0x90 [] ? request_threaded_irq+0x133/0x230 [] ? usb_add_hcd+0x4de/0x7e0 [] ? usb_hcd_pci_probe+0x164/0x3d0 [] ? local_pci_probe+0x17/0x20 [] ? pci_device_probe+0x101/0x120 [] ? driver_sysfs_add+0x62/0x90 [] ? driver_probe_device+0xa0/0x2a0 [] ? __driver_attach+0xab/0xb0 [] ? __driver_attach+0x0/0xb0 [] ? bus_for_each_dev+0x64/0x90 [] ? driver_attach+0x1e/0x20 [] ? bus_add_driver+0x1e8/0x2b0 [] ? driver_register+0x76/0x140 [] ? __pci_register_driver+0x56/0xd0 [] ? uhci_hcd_init+0x0/0xca [] ? uhci_hcd_init+0x0/0xca [] ? uhci_hcd_init+0x8c/0xca [] ? ohci_hcd_mod_init+0x61/0x90 [] ? do_one_initcall+0x3c/0x1d0 [] ? kernel_init+0x2a2/0x2fe [] ? child_rip+0xa/0x20 [] ? kernel_init+0x0/0x2fe [] ? child_rip+0x0/0x20 handlers: [] (usb_hcd_irq+0x0/0x90) Disabling IRQ #11 Refined TSC clocksource calibration: 2400.084 MHz. uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: UHCI Host Controller usb usb1: Manufacturer: Linux 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 uhci_hcd usb usb1: SerialNumber: 0000:00:01.2 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 serio: i8042 KBD port at 0x60,0x64 irq 1 serio: i8042 AUX port at 0x60,0x64 irq 12 mice: PS/2 mouse device common for all mice input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 rtc_cmos 00:01: RTC can wake from S4 rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 rtc0: alarms up to one day, 114 bytes nvram cpuidle: using governor ladder cpuidle: using governor menu EFI Variables Facility v0.08 2004-May-17 usbcore: registered new interface driver hiddev usbcore: registered new interface driver usbhid usbhid: v2.6:USB HID core driver TCP cubic registered Initializing XFRM netlink socket NET: Registered protocol family 17 registered taskstats version 1 rtc_cmos 00:01: setting system clock to 2013-05-26 06:40:09 UTC (1369550409) Initalizing network drop monitor service Freeing unused kernel memory: 1264k freed Write protecting the kernel read-only data: 10240k Freeing unused kernel memory: 904k freed Freeing unused kernel memory: 1672k freed Mounting proc filesystem Mounting sysfs filesystem Creating /dev Creating initial device nodes setfont: KDFONTOP: Invalid argument Free memory/Total memory (free %): 83508 / 114756 ( 72.7701 ) Loading sunrpc.ko module RPC: Registered named UNIX socket transport module. RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. Loading ipv6.ko module NET: Registered protocol family 10 lo: Disabled Privacy Extensions Loading ib_core.ko module Loading mii.ko module Loading i2c-core.ko module Loading jbd.ko module Loading mbcache.ko module Loading virtio_ring.ko module Loading virtio.ko module Loading pata_acpi.ko module Loading ata_generic.ko module Loading ata_piix.ko module scsi0 : ata_piix scsi1 : ata_piix ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 Loading dm-mod.ko module device-mapper: uevent: version 1.0.3 device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com Loading ib_uverbs.ko module Loading iw_cm.ko module Loading ib_addr.ko module Loading ib_mad.ko module Loading 8139too.ko module 8139too Fast Ethernet driver 0.9.28 8139too 0000:00:03.0: This (id 10ec:8139 rev 20) is an enhanced 8139C+ chip, use 8139cp Loading 8139cp.ko module 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10 eth0: RTL-8139C+ at 0xffffc9000006c000, 00:16:3e:33:7e:3f, IRQ 10 Loading i2c-piix4.ko module piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 Loading ext3.ko module Loading virtio_blk.ko module Loading virtio_pci.ko module virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 vda: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 vda1 vda2 vda4 ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 Loading dm-log.ko module Loading ib_umad.ko module Loading ib_sa.ko module Loading dm-region-hash.ko module Loading ib_cm.ko module Loading dm-mirror.ko module Loading ib_ipoib.ko module Loading ib_ucm.ko module Loading rdma_cm.ko module Loading rdma_ucm.ko module Waiting for required block device discovery Waiting for 2 vda-like device(s)...Found Creating Block Devices Creating block device loop0 Creating block device loop1 Creating block device loop2 Creating block device loop3 Creating block device loop4 Creating block device loop5 Creating block device loop6 Creating block device loop7 Creating block device ram0 Creating block device ram1 Creating block device ram10 Creating block device ram11 Creating block device ram12 Creating block device ram13 Creating block device ram14 Creating block device ram15 Creating block device ram2 Creating block device ram3 Creating block device ram4 Creating block device ram5 Creating block device ram6 Creating block device ram7 Creating block device ram8 Creating block device ram9 Creating block device vda vda: vda1 vda2 vda4 Saving to the local filesystem UUID=08cf7b2b-0dec-4c87-87dd-e442d5c4f95f e2fsck 1.42.7.wc1 (12-Apr-2013) /dev/vda1: recovering journal Clearing orphaned inode 254031 (uid=0, gid=0, mode=0100755, size=15256) Clearing orphaned inode 352362 (uid=0, gid=0, mode=0100755, size=40792) Clearing orphaned inode 352302 (uid=0, gid=0, mode=0100755, size=14584) Clearing orphaned inode 352280 (uid=0, gid=0, mode=0100755, size=113432) Clearing orphaned inode 352972 (uid=0, gid=0, mode=0100755, size=277704) Clearing orphaned inode 296720 (uid=0, gid=0, mode=0100755, size=1662832) Clearing orphaned inode 352322 (uid=0, gid=0, mode=0100755, size=265728) Clearing orphaned inode 296615 (uid=0, gid=0, mode=0100755, size=1284048) Clearing orphaned inode 295554 (uid=0, gid=0, mode=0100755, size=175288) Clearing orphaned inode 352331 (uid=0, gid=0, mode=0100755, size=14560) Clearing orphaned inode 352330 (uid=0, gid=0, mode=0100755, size=18720) Clearing orphaned inode 352329 (uid=0, gid=0, mode=0100755, size=240592) Clearing orphaned inode 352978 (uid=0, gid=0, mode=0100755, size=941920) Clearing orphaned inode 352333 (uid=0, gid=0, mode=0100755, size=12400) Clearing orphaned inode 352976 (uid=0, gid=0, mode=0100755, size=174840) Clearing orphaned inode 352980 (uid=0, gid=0, mode=0100755, size=43728) Clearing orphaned inode 352296 (uid=0, gid=0, mode=0100755, size=110960) Clearing orphaned inode 352968 (uid=0, gid=0, mode=0100755, size=10192) Clearing orphaned inode 352298 (uid=0, gid=0, mode=0100755, size=43832) Clearing orphaned inode 354221 (uid=0, gid=0, mode=0100755, size=10288) Clearing orphaned inode 352258 (uid=0, gid=0, mode=0100755, size=90784) Clearing orphaned inode 352338 (uid=0, gid=0, mode=0100755, size=122040) Clearing orphaned inode 352278 (uid=0, gid=0, mode=0100755, size=595688) Clearing orphaned inode 352320 (uid=0, gid=0, mode=0100755, size=88600) Clearing orphaned inode 352294 (uid=0, gid=0, mode=0100755, size=142464) Clearing orphaned inode 352393 (uid=0, gid=0, mode=0100755, size=55848) Clearing orphaned inode 352274 (uid=0, gid=0, mode=0100755, size=40400) Clearing orphaned inode 352262 (uid=0, gid=0, mode=0100755, size=383504) Clearing orphaned inode 352276 (uid=0, gid=0, mode=0100755, size=19536) Clearing orphaned inode 352324 (uid=0, gid=0, mode=0100755, size=113096) Clearing orphaned inode 352270 (uid=0, gid=0, mode=0100755, size=1916568) Clearing orphaned inode 352263 (uid=0, gid=0, mode=0100755, size=154464) Setting free inodes count to 1258852 (was 1258970) Setting free blocks count to 4715606 (was 4733036) /dev/vda1: clean, 51868/1310720 files, 527274/5242880 blocks kjournald starting. Commit interval 5 seconds EXT3-fs (vda1): using internal journal EXT3-fs (vda1): mounted filesystem with ordered data mode Free memory/Total memory (free %): 77784 / 114756 ( 67.7821 ) Loading SELINUX policy SELinux: Disabled at runtime. type=1404 audit(1369550415.305:2): selinux=0 auid=4294967295 ses=4294967295 Policy load failed, a relabel will be required on reboot Saving vmcore-dmesg.txt Saved vmcore-dmesg.txt Checking for memory holes : [ 0 %] Checking for memory holes : [100 %] Excluding unnecessary pages : [100 %] Excluding free pages : [100 %] Copying data : [ 0 %] Copying data : [ 8 %] Copying data : [ 17 %] Copying data : [ 28 %] Copying data : [ 47 %] Copying data : [ 58 %] Copying data : [ 74 %] Copying data : [ 86 %] Copying data : [100 %] Saving core complete umount: can't umount /mnt/selinux: Invalid argument Restarting system. machine restart Press any key to continue. Press any key to continue. Press any key to continue. Press any key to continue. Press any key to continue.  GNU GRUB version 0.97 (617K lower / 2094860K upper memory) +-------------------------------------------------------------------------+||||||||||||||||||||||||+-------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, 'e' to edit the commands before booting, 'a' to modify the kernel arguments before booting, or 'c' for a command-line.  CentOS (2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64)             The highlighted entry will be booted automatically in 5 seconds. The highlighted entry will be booted automatically in 4 seconds. The highlighted entry will be booted automatically in 3 seconds. The highlighted entry will be booted automatically in 2 seconds. The highlighted entry will be booted automatically in 1 seconds. Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 (jenkins@builder-4-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Thu May 23 10:02:55 PDT 2013 Command line: ro root=UUID=08cf7b2b-0dec-4c87-87dd-e442d5c4f95f rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200 crashkernel=auto SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls Disabled fast string operations BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009a400 (usable) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 000000007fec3000 (usable) BIOS-e820: 000000007ff00000 - 000000007fffd000 (usable) BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) DMI 2.4 present. SMBIOS version 2.4 @ 0xFDA50 last_pfn = 0x7fffd max_arch_pfn = 0x400000000 PAT not supported by CPU. init_memory_mapping: 0000000000000000-000000007fffd000 RAMDISK: 36c77000 - 37fefb5e ACPI: RSDP 00000000000fda20 00014 (v00 BOCHS ) ACPI: RSDT 000000007fffd740 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) ACPI: FACP 000000007ffffe40 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) ACPI: DSDT 000000007fffd900 024FD (v01 BXPC BXDSDT 00000001 INTL 20090123) ACPI: FACS 000000007ffffe00 00040 ACPI: SSDT 000000007fffd860 0009E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) ACPI: APIC 000000007fffd770 00078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) Setting APIC routing to flat. No NUMA configuration found Faking a node at 0000000000000000-000000007fffd000 Bootmem setup node 0 0000000000000000-000000007fffd000 NODE_DATA [000000000000a000 - 000000000003dfff] bootmap [000000000003e000 - 000000000004dfff] pages 10 (7 early reservations) ==> bootmem [0000000000 - 007fffd000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] #2 [0001000000 - 000201b0a4] TEXT DATA BSS ==> [0001000000 - 000201b0a4] #3 [0036c77000 - 0037fefb5e] RAMDISK ==> [0036c77000 - 0037fefb5e] #4 [000009a400 - 0000100000] BIOS reserved ==> [000009a400 - 0000100000] #5 [000201c000 - 000201c081] BRK ==> [000201c000 - 000201c081] #6 [0000008000 - 000000a000] PGTABLE ==> [0000008000 - 000000a000] found SMP MP-table at [ffff8800000fda70] fda70 Reserving 129MB of memory at 48MB for crashkernel (System RAM: 2047MB) kvm-clock: Using msrs 4b564d01 and 4b564d00 kvm-clock: cpu 0, msr 0:1c256c1, boot clock Zone PFN ranges: DMA 0x00000001 -> 0x00001000 DMA32 0x00001000 -> 0x00100000 Normal 0x00100000 -> 0x00100000 Movable zone start PFN for each node early_node_map[3] active PFN ranges 0: 0x00000001 -> 0x0000009a 0: 0x00000100 -> 0x0007fec3 0: 0x0007ff00 -> 0x0007fffd ACPI: PM-Timer IO Port: 0xb008 Setting APIC routing to flat. ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) Using ACPI (MADT) for SMP configuration information SMP: Allowing 1 CPUs, 0 hotplug CPUs PM: Registered nosave memory: 000000000009a000 - 00000000000f0000 PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 PM: Registered nosave memory: 000000007fec3000 - 000000007ff00000 Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) Booting paravirtualized kernel on KVM NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 PERCPU: Embedded 31 pages/cpu @ffff880002200000 s94616 r8192 d24168 u2097152 pcpu-alloc: s94616 r8192 d24168 u2097152 alloc=1*2097152 pcpu-alloc: [0] 0 kvm-clock: cpu 0, msr 0:22166c1, primary cpu clock kvm-stealtime: cpu 0, msr 220e840 Built 1 zonelists in Node order, mobility grouping on. Total pages: 516847 Policy zone: DMA32 Kernel command line: ro root=UUID=08cf7b2b-0dec-4c87-87dd-e442d5c4f95f rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200 crashkernel=129M@0M SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM PID hash table entries: 4096 (order: 3, 32768 bytes) Checking aperture... No AGP bridge found Memory: 1898648k/2097140k available (5222k kernel code, 656k absent, 197836k reserved, 7119k data, 1264k init) Hierarchical RCU implementation. NR_IRQS:33024 nr_irqs:256 Console: colour VGA+ 80x25 console [ttyS0] enabled allocated 8388608 bytes of page_cgroup please try 'cgroup_disable=memory' option if you don't want memory cgroups Detected 2400.084 MHz processor. Calibrating delay loop (skipped) preset value.. 4800.16 BogoMIPS (lpj=2400084) pid_max: default: 32768 minimum: 301 Security Framework initialized SELinux: Initializing. Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) Mount-cache hash table entries: 256 Initializing cgroup subsys ns Initializing cgroup subsys cpuacct Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls Initializing cgroup subsys blkio Initializing cgroup subsys perf_event Initializing cgroup subsys net_prio Disabled fast string operations mce: CPU supports 10 MCE banks alternatives: switching to unfair spinlock SMP alternatives: switching to UP code Freeing SMP alternatives: 35k freed ACPI: Core revision 20090903 ftrace: converting mcount calls to 0f 1f 44 00 00 ftrace: allocating 21438 entries in 85 pages Setting APIC routing to flat ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 CPU0: Intel QEMU Virtual CPU version (cpu64-rhel6) stepping 03 Performance Events: Broken PMU hardware detected, using software events only. NMI watchdog disabled (cpu0): hardware events not enabled Brought up 1 CPUs Total of 1 processors activated (4800.16 BogoMIPS). devtmpfs: initialized regulator: core version 0.5 NET: Registered protocol family 16 ACPI: bus type pci registered PCI: Using configuration type 1 for base access bio: create slab at 0 ACPI: Interpreter enabled ACPI: (supports S0 S5) ACPI: Using IOAPIC for interrupt routing ACPI: No dock devices found. HEST: Table not found. PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0 vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none vgaarb: loaded vgaarb: bridge control possible 0000:00:02.0 SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb PCI: Using ACPI for IRQ routing NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default Switching to clocksource kvm-clock pnp: PnP ACPI init ACPI: bus type pnp registered pnp: PnP ACPI: found 6 devices ACPI: ACPI bus type pnp unregistered NET: Registered protocol family 2 IP route cache hash table entries: 65536 (order: 7, 524288 bytes) TCP established hash table entries: 262144 (order: 10, 4194304 bytes) TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) TCP: Hash tables configured (established 262144 bind 65536) TCP reno registered NET: Registered protocol family 1 pci 0000:00:00.0: Limiting direct PCI/PCI transfers pci 0000:00:01.0: PIIX3: Enabling Passive Release pci 0000:00:01.0: Activating ISA DMA hang workarounds ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 pci 0000:00:01.2: PCI INT D disabled Trying to unpack rootfs image as initramfs... Freeing initrd memory: 19938k freed audit: initializing netlink socket (disabled) type=2000 audit(1369550465.459:1): initialized HugeTLB registered 2 MB page size, pre-allocated 0 pages VFS: Disk quotas dquot_6.5.2 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) msgmni has been set to 3747 alg: No test for stdrng (krng) ksign: Installing public key data Loading keyring - Added public key 30BD1F01F604D494 - User ID: Red Hat, Inc. (Kernel Module GPG key) - Added public key D4A26C9CCD09BEDA - User ID: Red Hat Enterprise Linux Driver Update Program Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered (default) io scheduler cfq registered pci_hotplug: PCI Hot Plug PCI Core version: 0.5 pciehp: PCI Express Hot Plug Controller Driver version: 0.4 acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 acpiphp: Slot [1] registered acpiphp: Slot [2] registered acpiphp: Slot [3] registered acpiphp: Slot [4] registered acpiphp: Slot [5] registered acpiphp: Slot [6] registered acpiphp: Slot [7] registered acpiphp: Slot [8] registered acpiphp: Slot [9] registered acpiphp: Slot [10] registered acpiphp: Slot [11] registered acpiphp: Slot [12] registered acpiphp: Slot [13] registered acpiphp: Slot [14] registered acpiphp: Slot [15] registered acpiphp: Slot [16] registered acpiphp: Slot [17] registered acpiphp: Slot [18] registered acpiphp: Slot [19] registered acpiphp: Slot [20] registered acpiphp: Slot [21] registered acpiphp: Slot [22] registered acpiphp: Slot [23] registered acpiphp: Slot [24] registered acpiphp: Slot [25] registered acpiphp: Slot [26] registered acpiphp: Slot [27] registered acpiphp: Slot [28] registered acpiphp: Slot [29] registered acpiphp: Slot [30] registered acpiphp: Slot [31] registered ipmi message handler version 39.2 IPMI System Interface driver. ipmi_si: Adding default-specified kcs state machine ipmi_si: Trying default-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Adding default-specified smic state machine ipmi_si: Trying default-specified smic state machine at i/o address 0xca9, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Adding default-specified bt state machine ipmi_si: Trying default-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Unable to find any System Interface(s) input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 ACPI: Power Button [PWRF] ERST: Table is not found! GHES: HEST is not enabled! Non-volatile memory driver v1.3 Linux agpgart interface v0.103 crash memory driver: version 1.1 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled erial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A brd: module loaded loop: module loaded input: Macintosh mouse button emulation as /devices/virtual/input/input1 Fixed MDIO Bus: probed ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver uhci_hcd: USB Universal Host Controller Interface driver uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 uhci_hcd 0000:00:01.2: UHCI Host Controller uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: UHCI Host Controller usb usb1: Manufacturer: Linux 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 uhci_hcd usb usb1: SerialNumber: 0000:00:01.2 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 serio: i8042 KBD port at 0x60,0x64 irq 1 serio: i8042 AUX port at 0x60,0x64 irq 12 mice: PS/2 mouse device common for all mice input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 rtc_cmos 00:01: RTC can wake from S4 rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 rtc0: alarms up to one day, 114 bytes nvram cpuidle: using governor ladder cpuidle: using governor menu EFI Variables Facility v0.08 2004-May-17 usbcore: registered new interface driver hiddev usbcore: registered new interface driver usbhid usbhid: v2.6:USB HID core driver TCP cubic registered Initializing XFRM netlink socket NET: Registered protocol family 17 registered taskstats version 1 rtc_cmos 00:01: setting system clock to 2013-05-26 06:41:06 UTC (1369550466) Initalizing network drop monitor service Freeing unused kernel memory: 1264k freed Write protecting the kernel read-only data: 10240k Freeing unused kernel memory: 904k freed Freeing unused kernel memory: 1672k freed dracut: dracut-004-303.el6 dracut: rd_NO_LUKS: removing cryptoluks activation dracut: rd_NO_LVM: removing LVM activation device-mapper: uevent: version 1.0.3 device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com udev: starting version 147 dracut: Starting plymouth daemon dracut: rd_NO_DM: removing DM RAID activation dracut: rd_NO_MD: removing MD RAID activation scsi0 : ata_piix scsi1 : ata_piix ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 %Ginput: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 vda: vda1 vda2 vda4 Refined TSC clocksource calibration: 2400.086 MHz. kjournald starting. Commit interval 5 seconds EXT3-fs (vda1): mounted filesystem with ordered data mode dracut: Mounted root filesystem /dev/vda1 SELinux: Disabled at runtime. type=1404 audit(1369550467.307:2): selinux=0 auid=4294967295 ses=4294967295 dracut: dracut: Switching root Welcome to CentOS Starting udev: udev: starting version 147 %Gpiix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 eth0: RTL-8139C+ at 0xffffc900009d8000, 00:16:3e:33:7e:3f, IRQ 11 8139too Fast Ethernet driver 0.9.28 microcode: CPU0 sig=0x6d3, pf=0x1, revision=0x1 platform microcode: firmware: requesting intel-ucode/06-0d-03 Microcode Update Driver: v2.00 , Peter Oruba [ OK ] Setting hostname client-26vm3.lab.whamcloud.com: [ OK ] Setting up Logical Volume Management: 3 logical volume(s) in volume group "lvm-MDS" now active [ OK ] Checking filesystems Checking all file systems. [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: clean, 51871/1310720 files, 539104/5242880 blocks [ OK ] Remounting root filesystem in read-write mode: EXT3-fs (vda1): using internal journal [ OK ] Mounting local filesystems: [ OK ] Enabling local filesystem quotas: [ OK ] Enabling /etc/fstab swaps: Adding 4128760k swap on /dev/vda2. Priority:-1 extents:1 across:4128760k [ OK ] Entering non-interactive startup Calling the system activity data collector (sadc): Starting monitoring for VG lvm-MDS: 3 logical volume(s) in volume group "lvm-MDS" monitored [ OK ] Loading OpenIB kernel modules:NET: Registered protocol family 10 lo: Disabled Privacy Extensions [ OK ] Bringing up loopback interface: [ OK ] Bringing up interface eth0: eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ OK ] Starting auditd: [ OK ] Starting system logger: [ OK ] Starting irqbalance: [ OK ] Starting rpcbind: [ OK ] Starting sssd: [ OK ] [ OK ] Starting NFS statd: [ OK ] Starting RPC idmapd: RPC: Registered named UNIX socket transport module. RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Starting kdump:[ OK ] Starting system message bus: [ OK ] Starting ibacm daemon:[ OK ] Setting up Logical Volume Management: 3 logical volume(s) in volume group "lvm-MDS" now active [ OK ] Checking network-attached filesystems [ OK ] Mounting other filesystems: kjournald starting. Commit interval 5 seconds EXT3-fs (dm-2): using internal journal EXT3-fs (dm-2): mounted filesystem with ordered data mode [ OK ] Starting acpi daemon: [ OK ] Starting HAL daemon: [ OK ] Retrigger failed udev events[ OK ] Loading autofs4: [ OK ] Starting automount: [ OK ] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Starting NFS services: [ OK ] Starting NFS quotas: [ OK ] Starting NFS mountd: [ OK ] Stopping RPC idmapd: [ OK ] Starting RPC idmapd: [ OK ] Starting NFS daemon: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory NFSD: starting 90-second grace period [ OK ] Starting MUNGE: munged[ OK ] Starting sshd: [ OK ] Starting xinetd: [ OK ] Starting ntpd: [ OK ] Starting sendmail: [ OK ] Starting sm-client: [ OK ] Starting crond: [ OK ] Starting atd: [ OK ] CentOS release 6.4 (Final) Kernel 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 on an x86_64 client-26vm3.lab.whamcloud.com login: Slow work thread pool: Starting up Slow work thread pool: Ready FS-Cache: Loaded Registering the id_resolver key type FS-Cache: Netfs 'nfs' registered for caching SysRq : Show State task PC stack pid father init S 0000000000000000 0 1 0 0x00000000 ffff88007e487908 0000000000000082 ffff88007e487878 ffffffff8105230c ffff88007e487878 0000000000000082 000000107e48ba00 00000000000616d0 ffff88007e485ab8 ffff88007e487fd8 000000000000fb88 ffff88007e485ab8 Call Trace: [] ? check_preempt_curr+0x7c/0x90 [] ? get_page_from_freelist+0x3d1/0x830 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? handle_pte_fault+0x2cd/0xb50 [] ? d_lookup+0x3c/0x60 [] ? mutex_lock+0x1e/0x50 [] ? pipe_read+0x2a7/0x4e0 [] core_sys_select+0x18a/0x2c0 [] ? security_task_wait+0x16/0x20 [] ? wait_consider_task+0x9d/0xb20 [] ? remove_wait_queue+0x3c/0x50 [] ? do_wait+0x17f/0x240 [] ? pvclock_clocksource_read+0x58/0xd0 [] sys_select+0x47/0x110 [] ? posix_ktime_get_ts+0x11/0x20 [] system_call_fastpath+0x16/0x1b kthreadd S 0000000000000000 0 2 0 0x00000000 ffff88007e48ded0 0000000000000046 ffff88007e48df40 ffffffff8100c062 ffff88007e484aa0 ffff88007e484aa0 ffffffff81eafb88 ffff88007d0579a0 ffff88007e485058 ffff88007e48dfd8 000000000000fb88 ffff88007e485058 Call Trace: [] ? kernel_thread+0x82/0xe0 [] ? kthread+0x0/0xa0 [] kthreadd+0x1b5/0x1c0 [] child_rip+0xa/0x20 [] ? kthreadd+0x0/0x1c0 [] ? child_rip+0x0/0x20 migration/0 S 0000000000000000 0 3 2 0x00000000 ffff88007e491e70 0000000000000046 ffff88007e485538 0000000000000000 0000000000491e20 ffff8800022142c0 ffff88007e491e10 ffff880002216740 ffff88007e4845f8 ffff88007e491fd8 000000000000fb88 ffff88007e4845f8 Call Trace: [] ? thread_return+0x4e/0x76e [] migration_thread+0x265/0x2e0 [] ? migration_thread+0x0/0x2e0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ksoftirqd/0 S 0000000000000000 0 4 2 0x00000000 ffff88007e495ea0 0000000000000046 0000000000000000 0000000000000000 0000000000016740 0000000000000000 0000000000000000 ffffffff81ac2700 ffff88007e493af8 ffff88007e495fd8 000000000000fb88 ffff88007e493af8 Call Trace: [] ? default_wake_function+0x12/0x20 [] ksoftirqd+0xd5/0x110 [] ? ksoftirqd+0x0/0x110 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 migration/0 S 0000000000000000 0 5 2 0x00000000 ffff88007e497df0 0000000000000046 ffff88007e497d50 ffffffff81a97840 0000000000000086 ffffffff81aa5280 ffff88007e497d90 ffffffff810b3b18 ffff88007e493098 ffff88007e497fd8 000000000000fb88 ffff88007e493098 Call Trace: [] ? module_refcount+0x58/0x70 [] ? stop_machine_cpu_stop+0x0/0xf0 [] cpu_stopper_thread+0x125/0x1b0 [] ? thread_return+0x4e/0x76e [] ? default_wake_function+0x12/0x20 [] ? cpu_stopper_thread+0x0/0x1b0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 watchdog/0 S 0000000000000000 0 6 2 0x00000000 ffff88007e49bea0 0000000000000046 ffff88007e49be00 ffffffff8103b8ac ffff88007e49be10 ffffffff81012b69 ffff88007e49be40 ffffffff8109dc05 ffff88007e492638 ffff88007e49bfd8 000000000000fb88 ffff88007e492638 Call Trace: [] ? kvm_clock_read+0x1c/0x20 [] ? sched_clock+0x9/0x10 [] ? sched_clock_local+0x25/0x90 [] watchdog+0x9a/0xd0 [] ? watchdog+0x0/0xd0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 events/0 S 0000000000000000 0 7 2 0x00000000 ffff88007e4c7e30 0000000000000046 ffff88007e4c7dc0 ffffffff81091a89 ffff88007e4c7da0 ffffffff81082128 ffff88007e4c7de0 ffffffff81b1b964 ffff88007e4c5ab8 ffff88007e4c7fd8 000000000000fb88 ffff88007e4c5ab8 Call Trace: [] ? queue_delayed_work_on+0xb9/0x120 [] ? add_timer+0x18/0x30 [] ? prepare_to_wait+0x4e/0x80 [] ? neigh_periodic_work+0x0/0x1b0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 cgroup S 0000000000000000 0 8 2 0x00000000 ffff88007e4c9e30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007e4c9db0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007e4c5058 ffff88007e4c9fd8 000000000000fb88 ffff88007e4c5058 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 khelper S 0000000000000000 0 9 2 0x00000000 ffff88007e4cfe30 0000000000000046 ffffffff8108f8e0 0000000000000001 ffff88007e4cfe30 ffff88007cd50240 0000000000000000 0000000000000000 ffff88007e4c45f8 ffff88007e4cffd8 000000000000fb88 ffff88007e4c45f8 Call Trace: [] ? __call_usermodehelper+0x0/0xd0 [] ? prepare_to_wait+0x4e/0x80 [] ? __call_usermodehelper+0x0/0xd0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 netns S 0000000000000000 0 10 2 0x00000000 ffff88007e4f3e30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007e4f3db0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007e4f1af8 ffff88007e4f3fd8 000000000000fb88 ffff88007e4f1af8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 async/mgr S 0000000000000000 0 11 2 0x00000000 ffff88007e4f5e70 0000000000000046 0000000000000000 0000000000000000 ffff88007e4f5df0 ffffffff8105230c ffff88007e4f5df0 0000000000000282 ffff88007e4f1098 ffff88007e4f5fd8 000000000000fb88 ffff88007e4f1098 Call Trace: [] ? check_preempt_curr+0x7c/0x90 [] async_manager_thread+0xc5/0x100 [] ? default_wake_function+0x0/0x20 [] ? async_manager_thread+0x0/0x100 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 pm S 0000000000000000 0 12 2 0x00000000 ffff88007e4f9e30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007e4f9db0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007e4f0638 ffff88007e4f9fd8 000000000000fb88 ffff88007e4f0638 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 sync_supers S 0000000000000000 0 13 2 0x00000000 ffff88007e55bec0 0000000000000046 0000000000000000 ffffffff81ac2700 ffff88007e55bee0 ffffffff8150df90 0000000000000001 ffff88007e487de8 ffff88007e559ab8 ffff88007e55bfd8 000000000000fb88 ffff88007e559ab8 Call Trace: [] ? thread_return+0x4e/0x76e [] ? bdi_sync_supers+0x0/0x60 [] bdi_sync_supers+0x3b/0x60 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 bdi-default S 0000000000000000 0 14 2 0x00000000 ffff88007e55ddd0 0000000000000046 ffff8800022167d8 ffff8800022167a8 ffffffff81ac16d0 0000000000000282 ffff88007e55dd70 ffffffff81080fcc ffff88007e559058 ffff88007e55dfd8 000000000000fb88 ffff88007e559058 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] bdi_forker_task+0x263/0x310 [] ? bdi_forker_task+0x0/0x310 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kintegrityd/0 S 0000000000000000 0 15 2 0x00000000 ffff88007e561e30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007e561db0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007e5585f8 ffff88007e561fd8 000000000000fb88 ffff88007e5585f8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kblockd/0 S 0000000000000000 0 16 2 0x00000000 ffff88007e56be30 0000000000000046 000000037e485548 ffff880037b34e68 ffff880037b34e68 ffff880037b34e68 ffff880037b34e68 ffffffff81256ff0 ffff88007e569af8 ffff88007e56bfd8 000000000000fb88 ffff88007e569af8 Call Trace: [] ? blk_unplug_work+0x0/0x70 [] ? prepare_to_wait+0x4e/0x80 [] ? blk_unplug_work+0x0/0x70 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kacpid S 0000000000000000 0 17 2 0x00000000 ffff88007e56de30 0000000000000046 ffff880002216740 0000000000000282 ffff88007e56de10 ffffffff8106a586 ffff88007e56de10 0000000000000286 ffff88007e569098 ffff88007e56dfd8 000000000000fb88 ffff88007e569098 Call Trace: [] ? set_cpus_allowed_ptr+0xb6/0x1d0 [] ? prepare_to_wait+0x4e/0x80 [] ? bind_to_cpu0+0x0/0x32 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kacpi_notify S 0000000000000000 0 18 2 0x00000000 ffff88007e681e30 0000000000000046 ffff880002216740 0000000000000282 ffff88007e681e10 ffffffff8106a586 ffff88007e681e10 0000000000000286 ffff88007e568638 ffff88007e681fd8 000000000000fb88 ffff88007e568638 Call Trace: [] ? set_cpus_allowed_ptr+0xb6/0x1d0 [] ? prepare_to_wait+0x4e/0x80 [] ? bind_to_cpu0+0x0/0x32 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kacpi_hotplug S 0000000000000000 0 19 2 0x00000000 ffff88007e685e30 0000000000000046 ffff880002216740 0000000000000282 ffff88007e685e10 ffffffff8106a586 ffff88007e685e10 0000000000000286 ffff88007e683ab8 ffff88007e685fd8 000000000000fb88 ffff88007e683ab8 Call Trace: [] ? set_cpus_allowed_ptr+0xb6/0x1d0 [] ? prepare_to_wait+0x4e/0x80 [] ? bind_to_cpu0+0x0/0x32 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ata/0 S 0000000000000000 0 20 2 0x00000000 ffff88007e799e30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007e799db0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007e683058 ffff88007e799fd8 000000000000fb88 ffff88007e683058 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ata_aux S 0000000000000000 0 21 2 0x00000000 ffff88007e79de30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007e79ddb0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007e6825f8 ffff88007e79dfd8 000000000000fb88 ffff88007e6825f8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ksuspend_usbd S 0000000000000000 0 22 2 0x00000000 ffff88007e7a1e30 0000000000000046 0000000000000000 0000000000000402 ffff88007e7a1df0 ffffffff813bb08c ffff88007e485538 ffff8800022167a8 ffff88007e79faf8 ffff88007e7a1fd8 000000000000fb88 ffff88007e79faf8 Call Trace: [] ? usb_suspend_both+0x1bc/0x340 [] ? prepare_to_wait+0x4e/0x80 [] ? usb_autosuspend_work+0x0/0x20 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 khubd S 0000000000000000 0 23 2 0x00000000 ffff88007e7a5d60 0000000000000046 a2d49a8353aa354a 80d3ea3d4ba87536 0000000000000000 000004008103c7b8 ffff88007e7a3680 ffff88007978dc00 ffff88007e79f098 ffff88007e7a5fd8 000000000000fb88 ffff88007e79f098 Call Trace: [] ? prepare_to_wait+0x4e/0x80 [] hub_thread+0x379/0x1a00 [] ? __dequeue_entity+0x30/0x50 [] ? autoremove_wake_function+0x0/0x40 [] ? hub_thread+0x0/0x1a00 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kseriod S 0000000000000000 0 24 2 0x00000000 ffff88007e7a9e50 0000000000000046 ffff88007e7a9dd0 ffffffff8109c99f ffff88007e7a9e10 ffff88007e70dcf8 ffff88007e7a9e30 0000000000000286 ffff88007e79e638 ffff88007e7a9fd8 000000000000fb88 ffff88007e79e638 Call Trace: [] ? up+0x2f/0x50 [] ? prepare_to_wait+0x4e/0x80 [] serio_thread+0x39a/0x450 [] ? autoremove_wake_function+0x0/0x40 [] ? serio_thread+0x0/0x450 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 md/0 S 0000000000000000 0 25 2 0x00000000 ffff88007e7ade30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007e7addb0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007e7abab8 ffff88007e7adfd8 000000000000fb88 ffff88007e7abab8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 md_misc/0 S 0000000000000000 0 26 2 0x00000000 ffff88007e7afe30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007e7afdb0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007e7ab058 ffff88007e7affd8 000000000000fb88 ffff88007e7ab058 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 khungtaskd S 0000000000000000 0 27 2 0x00000000 ffff8800794fddd0 0000000000000046 ffff88007e485500 0000000000000000 00000000ffffffff 0000000000000282 ffff8800794fdd70 ffffffff81080fcc ffff88007e7aa5f8 ffff8800794fdfd8 000000000000fb88 ffff88007e7aa5f8 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] schedule_timeout_interruptible+0x1e/0x20 [] watchdog+0x52/0x220 [] ? watchdog+0x0/0x220 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kswapd0 S 0000000000000000 0 28 2 0x00000000 ffff880079561e20 0000000000000046 ffff880079561e20 ffffffff8150ed8a ffff880002052000 0000000000000282 0000000000000000 dead000000200200 ffff8800794ffaf8 ffff880079561fd8 000000000000fb88 ffff8800794ffaf8 Call Trace: [] ? schedule_timeout+0x19a/0x2e0 [] ? zone_watermark_ok_safe+0xad/0xc0 [] ? calculate_normal_threshold+0x0/0x60 [] kswapd+0x311/0x3c0 [] ? autoremove_wake_function+0x0/0x40 [] ? kswapd+0x0/0x3c0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ksmd S 0000000000000000 0 29 2 0x00000000 ffff880079563e20 0000000000000046 ffff8800794feae0 ffff8800794feae0 ffff880079563db0 ffffffff81065ceb ffff880079563db0 ffff8800794feae0 ffff8800794ff098 ffff880079563fd8 000000000000fb88 ffff8800794ff098 Call Trace: [] ? enqueue_task_fair+0xfb/0x100 [] ? prepare_to_wait+0x4e/0x80 [] ksm_scan_thread+0x81b/0xd80 [] ? autoremove_wake_function+0x0/0x40 [] ? ksm_scan_thread+0x0/0xd80 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 khugepaged S 0000000000000000 0 30 2 0x00000000 ffff88007956bd90 0000000000000046 ffff88007956bd90 ffffffff8150ed8a ffff88007956bd20 0000000000000282 0000000000000000 dead000000200200 ffff8800794fe638 ffff88007956bfd8 000000000000fb88 ffff8800794fe638 Call Trace: [] ? schedule_timeout+0x19a/0x2e0 [] ? prepare_to_wait+0x4e/0x80 [] khugepaged+0xaf2/0x1310 [] ? __switch_to+0xd0/0x320 [] ? thread_return+0x4e/0x76e [] ? autoremove_wake_function+0x0/0x40 [] ? khugepaged+0x0/0x1310 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 aio/0 S 0000000000000000 0 31 2 0x00000000 ffff88007956fe30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007956fdb0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007956dab8 ffff88007956ffd8 000000000000fb88 ffff88007956dab8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 crypto/0 S 0000000000000000 0 32 2 0x00000000 ffff88007965be30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007965bdb0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007956d058 ffff88007965bfd8 000000000000fb88 ffff88007956d058 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kthrotld/0 S 0000000000000000 0 37 2 0x00000000 ffff88007966fe30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff88007966fdb0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007966d098 ffff88007966ffd8 000000000000fb88 ffff88007966d098 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kpsmoused S 0000000000000000 0 39 2 0x00000000 ffff8800797a5e30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff8800797a5db0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007966daf8 ffff8800797a5fd8 000000000000fb88 ffff88007966daf8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 usbhid_resume S 0000000000000000 0 40 2 0x00000000 ffff8800797d9e30 0000000000000046 ffff88007e485548 ffff8800022167a8 ffff8800797d9db0 ffffffff810572f0 ffff88007e485538 ffff8800022167a8 ffff88007956c5f8 ffff8800797d9fd8 000000000000fb88 ffff88007956c5f8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kstriped S 0000000000000000 0 70 2 0x00000000 ffff880037c3de30 0000000000000046 ffffe8ffffc00040 ffff880037c3dee0 ffffffff81510d85 ffff88007e484aa0 ffffe8ffffc00040 ffffffff81090970 ffff880037c17ab8 ffff880037c3dfd8 000000000000fb88 ffff880037c17ab8 Call Trace: [] ? page_fault+0x25/0x30 [] ? worker_thread+0x0/0x2a0 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 scsi_eh_0 S 0000000000000000 0 250 2 0x00000000 ffff880037fc7e30 0000000000000046 000000000000000f 0000000000000282 ffff880037fc7df0 ffffffff81393ad8 ffff880037fc7df0 ffffffff81393fda ffff880037ffbaf8 ffff880037fc7fd8 000000000000fb88 ffff880037ffbaf8 Call Trace: [] ? ata_scsi_port_error_handler+0x348/0x710 [] ? ata_scsi_cmd_error_handler+0x13a/0x190 [] scsi_error_handler+0x96/0x6b0 [] ? default_wake_function+0x12/0x20 [] ? scsi_error_handler+0x0/0x6b0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 scsi_eh_1 S 0000000000000000 0 253 2 0x00000000 ffff8800379d1e30 0000000000000046 000000000000000f 0000000000000282 ffff8800379d1df0 ffffffff81393ad8 ffff8800379d1df0 ffffffff81393fda ffff8800379aa638 ffff8800379d1fd8 000000000000fb88 ffff8800379aa638 Call Trace: [] ? ata_scsi_port_error_handler+0x348/0x710 [] ? ata_scsi_cmd_error_handler+0x13a/0x190 [] scsi_error_handler+0x96/0x6b0 [] ? default_wake_function+0x12/0x20 [] ? scsi_error_handler+0x0/0x6b0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 virtio-blk S 0000000000000000 0 298 2 0x00000000 ffff880037b15e30 0000000000000046 ffff880037870b28 ffff8800022167a8 ffff880037b15db0 ffffffff810572f0 ffff880037870b18 ffff8800022167a8 ffff880037b11098 ffff880037b15fd8 000000000000fb88 ffff880037b11098 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kjournald S 0000000000000000 0 377 2 0x00000000 ffff880037b6be60 0000000000000046 ffff880074a34024 ffff880000000000 0000000000000000 0000000000000282 ffff880037b6be00 ffffffff81080fcc ffff880037ac65f8 ffff880037b6bfd8 000000000000fb88 ffff880037ac65f8 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] ? prepare_to_wait+0x4e/0x80 [] kjournald+0x22a/0x250 [jbd] [] ? autoremove_wake_function+0x0/0x40 [] ? kjournald+0x0/0x250 [jbd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 udevd S 0000000000000000 0 459 1 0x00000000 ffff88007dfb7a58 0000000000000086 ffff88007dfb79b8 ffffffff81063322 ffff88007dfb79f8 ffffffff811973e6 0000000000000000 ffff88007d5ba040 ffff880037ffa638 ffff88007dfb7fd8 000000000000fb88 ffff880037ffa638 Call Trace: [] ? default_wake_function+0x12/0x20 [] ? pollwake+0x56/0x60 [] schedule_hrtimeout_range+0x13d/0x160 [] ? __pollwait+0x75/0xf0 [] ? add_wait_queue+0x46/0x60 [] ? unix_dgram_poll+0x127/0x1c0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? autoremove_wake_function+0x0/0x40 [] ? generic_file_aio_write+0xba/0x100 [] ? do_sync_write+0xfa/0x140 [] ? sys_recvfrom+0xee/0x180 [] ? vfs_write+0x132/0x1a0 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b flush-252:0 S 0000000000000000 0 655 2 0x00000000 ffff880037913da0 0000000000000046 ffff880037913d30 ffff8800022167a8 ffff8800022167d8 0000000000000282 ffff880037b35118 ffff880037b35108 ffff88007d5bb058 ffff880037913fd8 000000000000fb88 ffff88007d5bb058 Call Trace: [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] schedule_timeout_interruptible+0x1e/0x20 [] bdi_writeback_task+0xa0/0x1b0 [] ? bit_waitqueue+0x17/0xd0 [] ? bdi_start_fn+0x0/0x100 [] bdi_start_fn+0x86/0x100 [] ? bdi_start_fn+0x0/0x100 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 vballoon S 0000000000000000 0 664 2 0x00000000 ffff88007ce67e40 0000000000000046 ffff88007ce67db0 ffffffff810572f0 ffff880037f4ead8 ffff8800022167a8 ffff88007ce67e10 ffffffff810096f0 ffff880037ae1af8 ffff88007ce67fd8 000000000000fb88 ffff880037ae1af8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? __switch_to+0xd0/0x320 [] ? vp_get+0x50/0x70 [virtio_pci] [] balloon+0x204/0x260 [virtio_balloon] [] ? autoremove_wake_function+0x0/0x40 [] ? balloon+0x0/0x260 [virtio_balloon] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kdmflush S 0000000000000000 0 781 2 0x00000000 ffff88007d07de30 0000000000000046 ffff880037b42ae8 ffff8800022167a8 ffff88007d07ddb0 ffffffff810572f0 ffff880037b42ad8 ffff8800022167a8 ffff8800378885f8 ffff88007d07dfd8 000000000000fb88 ffff8800378885f8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? dm_wq_work+0x0/0xf0 [dm_mod] [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 udevd S 0000000000000000 0 784 459 0x00000000 ffff88007cf15a08 0000000000000086 ffff880037f4d5c0 ffff8800022167a8 0000000000002901 0000000000000000 ffff88007cf159b8 ffffffff8127dcf5 ffff88007d5ba5f8 ffff88007cf15fd8 000000000000fb88 ffff88007d5ba5f8 Call Trace: [] ? rb_insert_color+0x125/0x160 [] ? enqueue_entity+0x125/0x410 [] ? wakeup_preempt_entity+0x47/0x50 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] ? try_to_wake_up+0x24e/0x3e0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? default_wake_function+0x12/0x20 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? sock_def_readable+0x44/0x80 [] ? unix_dgram_sendmsg+0x5e3/0x6b0 [] ? sock_sendmsg+0x123/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? __sys_sendmsg+0x406/0x420 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? sys_sendto+0x139/0x190 [] sys_ppoll+0x4c/0x180 [] ? sys_sendmsg+0x49/0x90 [] system_call_fastpath+0x16/0x1b kdmflush S 0000000000000000 0 785 2 0x00000000 ffff88007cf45e30 0000000000000046 ffff880037b42ae8 ffff8800022167a8 ffff88007cf45db0 ffffffff810572f0 ffff880037b42ad8 ffff8800022167a8 ffff880037ae0638 ffff88007cf45fd8 000000000000fb88 ffff880037ae0638 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? dm_wq_work+0x0/0xf0 [dm_mod] [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 udevd S 0000000000000000 0 786 459 0x00000000 ffff88007b371a08 0000000000000082 ffff88007b371988 ffffffff81055d4d ffff880037b43500 0000000000000000 0000000000000001 0000000000000082 ffff880037ae1098 ffff88007b371fd8 000000000000fb88 ffff880037ae1098 Call Trace: [] ? task_rq_lock+0x5d/0xa0 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] ? try_to_wake_up+0x24e/0x3e0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? default_wake_function+0x12/0x20 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? sock_def_readable+0x44/0x80 [] ? unix_dgram_sendmsg+0x5e3/0x6b0 [] ? sock_sendmsg+0x123/0x150 [] ? autoremove_wake_function+0x0/0x40 [] ? __sys_sendmsg+0x406/0x420 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? sys_sendto+0x139/0x190 [] sys_ppoll+0x4c/0x180 [] ? sys_sendmsg+0x49/0x90 [] system_call_fastpath+0x16/0x1b kdmflush S 0000000000000000 0 787 2 0x00000000 ffff880037445e30 0000000000000046 ffff880037b42ae8 ffff8800022167a8 ffff880037445db0 ffffffff810572f0 ffff880037b42ad8 ffff8800022167a8 ffff880037f4e5f8 ffff880037445fd8 000000000000fb88 ffff880037f4e5f8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? dm_wq_work+0x0/0xf0 [dm_mod] [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kauditd S 0000000000000000 0 916 2 0x00000000 ffff88007d829e70 0000000000000046 ffff880037db2e80 ffff88007cf46ae0 0000000000000001 ffff880037db2e80 ffffffff810d378b ffff88007cf46ae0 ffff88007cf47098 ffff88007d829fd8 000000000000fb88 ffff88007cf47098 Call Trace: [] ? kauditd_send_skb+0x3b/0x90 [] kauditd_thread+0x149/0x180 [] ? default_wake_function+0x0/0x20 [] ? kauditd_thread+0x0/0x180 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 infiniband/0 S 0000000000000000 0 959 2 0x00000000 ffff880037f67e30 0000000000000046 ffff88007cf460c8 ffff8800022167a8 ffff880037f67db0 ffffffff810572f0 ffff88007cf460b8 ffff8800022167a8 ffff88007cf47af8 ffff880037f67fd8 000000000000fb88 ffff88007cf47af8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ib_mcast S 0000000000000000 0 966 2 0x00000000 ffff880037f65e30 0000000000000046 ffff88007a8c6088 ffff8800022167a8 ffff880037f65db0 ffffffff810572f0 ffff88007a8c6078 ffff8800022167a8 ffff88007d5bbab8 ffff880037f65fd8 000000000000fb88 ffff88007d5bbab8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ib_addr S 0000000000000000 0 970 2 0x00000000 ffff8800378b5e30 0000000000000046 ffff880037c3e088 ffff8800022167a8 ffff8800378b5db0 ffffffff810572f0 ffff880037c3e078 ffff8800022167a8 ffff88007cf46638 ffff8800378b5fd8 000000000000fb88 ffff88007cf46638 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? process_req+0x0/0x1a0 [ib_addr] [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 iw_cm_wq S 0000000000000000 0 974 2 0x00000000 ffff8800378bfe30 0000000000000046 ffff880037c3e088 ffff8800022167a8 ffff8800378bfdb0 ffffffff810572f0 ffff880037c3e078 ffff8800022167a8 ffff88007a8c65f8 ffff8800378bffd8 000000000000fb88 ffff88007a8c65f8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ib_cm/0 S 0000000000000000 0 978 2 0x00000000 ffff88007df49e30 0000000000000046 ffff880037c3f548 ffff8800022167a8 ffff88007df49db0 ffffffff810572f0 ffff880037c3f538 ffff8800022167a8 ffff880037b43058 ffff88007df49fd8 000000000000fb88 ffff880037b43058 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 rdma_cm S 0000000000000000 0 982 2 0x00000000 ffff8800378bde30 0000000000000046 ffff880037c3f548 ffff8800022167a8 ffff8800378bddb0 ffffffff810572f0 ffff880037c3f538 ffff8800022167a8 ffff880037c3e5f8 ffff8800378bdfd8 000000000000fb88 ffff880037c3e5f8 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 ipoib S 0000000000000000 0 998 2 0x00000000 ffff880037807e30 0000000000000046 ffff880037edb588 ffff8800022167a8 ffff880037807db0 ffffffff810572f0 ffff880037edb578 ffff8800022167a8 ffff880037edb098 ffff880037807fd8 000000000000fb88 ffff880037edb098 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 auditd S 0000000000000000 0 1196 1 0x00000000 ffff88007d783da8 0000000000000082 ffff88007d783d98 ffffffff8109bb05 ffff880000000000 0000000100000282 ffff88007d783d88 ffffffff8106317e ffff88007966c638 ffff88007d783fd8 000000000000fb88 ffff88007966c638 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? try_to_wake_up+0x24e/0x3e0 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] ep_poll+0x2ad/0x330 [] ? default_wake_function+0x0/0x20 [] ? ktime_get_ts+0xa9/0xe0 [] sys_epoll_wait+0xc5/0xe0 [] system_call_fastpath+0x16/0x1b auditd S 0000000000000000 0 1197 1 0x00000000 ffff88007d785bf8 0000000000000082 ffff88007d785b68 ffffffff81119ec7 00000000000000cf ffffea0001984a08 ffff88007d785bd8 ffffffffa008b690 ffff88007b37d098 ffff88007d785fd8 000000000000fb88 ffff88007b37d098 Call Trace: [] ? unlock_page+0x27/0x30 [] ? ext3_ordered_write_end+0x100/0x1c0 [ext3] [] futex_wait_queue_me+0xb9/0xf0 [] futex_wait+0x1f8/0x380 [] ? futex_wake+0x10e/0x120 [] do_futex+0x121/0xb60 [] ? statfs_by_dentry+0x74/0xa0 [] ? vfs_statfs+0x1b/0xb0 [] ? do_statfs_native+0x20/0xb0 [] sys_futex+0x7b/0x170 [] system_call_fastpath+0x16/0x1b rsyslogd S 0000000000000000 0 1212 1 0x00000080 ffff88007ccf5908 0000000000000086 ffff88007ccf58f8 ffffffff8109bb05 ffff880000000000 0000000100000000 ffff88007ccf4000 ffffea0001a71ce8 ffff88007dfc5af8 ffff88007ccf5fd8 000000000000fb88 ffff88007dfc5af8 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? zone_statistics+0x99/0xc0 [] ? __pollwait+0x0/0xf0 [] ? zone_statistics+0x99/0xc0 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? find_get_page+0x1e/0xa0 [] ? filemap_fault+0xd3/0x500 [] ? unlock_page+0x27/0x30 [] ? __do_fault+0x469/0x530 [] ? handle_pte_fault+0xf7/0xb50 [] ? kvm_clock_read+0x1c/0x20 [] ? sched_clock_local+0x25/0x90 [] ? pvclock_clocksource_read+0x58/0xd0 [] core_sys_select+0x18a/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] sys_select+0x47/0x110 [] system_call_fastpath+0x16/0x1b rs:main Q:Reg S 0000000000000000 0 1213 1 0x00000080 ffff88007ccf7bf8 0000000000000086 ffff88007ccf7b68 ffffffff81119ec7 000000000000009c ffffea000199b3d8 ffff88007ccf7bd8 ffffffffa008b690 ffff880037f4f058 ffff88007ccf7fd8 000000000000fb88 ffff880037f4f058 Call Trace: [] ? unlock_page+0x27/0x30 [] ? ext3_ordered_write_end+0x100/0x1c0 [ext3] [] futex_wait_queue_me+0xb9/0xf0 [] futex_wait+0x1f8/0x380 [] ? __sb_end_write+0x3d/0x70 [] ? generic_file_aio_write+0xba/0x100 [] do_futex+0x121/0xb60 [] ? user_path_at+0x62/0xa0 [] ? autoremove_wake_function+0x0/0x40 [] sys_futex+0x7b/0x170 [] ? audit_syscall_entry+0x1d7/0x200 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b rsyslogd S 0000000000000000 0 1215 1 0x00000080 ffff88007cdfb908 0000000000000086 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880037f4fab8 ffff88007cdfbfd8 000000000000fb88 ffff880037f4fab8 Call Trace: [] ? pvclock_clocksource_read+0x58/0xd0 [] schedule_hrtimeout_range+0x13d/0x160 [] ? __pollwait+0x75/0xf0 [] ? unix_dgram_poll+0x127/0x1c0 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? zone_statistics+0x99/0xc0 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? sock_recvmsg+0x133/0x160 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? enqueue_entity+0x125/0x410 [] ? check_preempt_wakeup+0x1a4/0x260 [] ? enqueue_task_fair+0xfb/0x100 [] ? check_preempt_curr+0x7c/0x90 [] ? try_to_wake_up+0x24e/0x3e0 [] core_sys_select+0x18a/0x2c0 [] ? pvclock_clocksource_read+0x58/0xd0 [] sys_select+0x47/0x110 [] ? audit_syscall_entry+0x1d7/0x200 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b rsyslogd S 0000000000000000 0 1216 1 0x00000080 ffff88007c469df8 0000000000000086 ffff88007b37aec0 0000000000000282 ffff88007b37aec0 ffff880037f4eaa0 ffff88007ccf7c98 00007f84bbf50420 ffff88007b37daf8 ffff88007c469fd8 000000000000fb88 ffff88007b37daf8 Call Trace: [] ? prepare_to_wait+0x4e/0x80 [] do_syslog+0x461/0x4c0 [] ? autoremove_wake_function+0x0/0x40 [] kmsg_read+0x32/0x60 [] proc_reg_read+0x7e/0xc0 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b rpcbind S 0000000000000000 0 1254 1 0x00000080 ffff880037af3a58 0000000000000082 ffff880037af3a48 ffffffff8109bb05 0000000000000000 0000000181b16540 ffff880037af39e8 ffffffff81484e0b ffff880037829ab8 ffff880037af3fd8 000000000000fb88 ffff880037829ab8 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? ip_send_skb+0x1b/0x80 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? anon_vma_chain_free+0x18/0x20 [] ? unlink_anon_vmas+0x71/0xd0 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? poll_select_set_timeout+0x8d/0xa0 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b sssd S 0000000000000000 0 1266 1 0x00000080 ffff880037ab7da8 0000000000000086 ffff880037ab7d98 ffffffff8109bb05 ffff880000000000 000000018109ac0b 0000080000000040 ffff88007c9ba640 ffff880037c165f8 ffff880037ab7fd8 000000000000fb88 ffff880037c165f8 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] ep_poll+0x2ad/0x330 [] ? default_wake_function+0x0/0x20 [] sys_epoll_wait+0xc5/0xe0 [] system_call_fastpath+0x16/0x1b sssd_be S 0000000000000000 0 1267 1266 0x00000080 ffff880037ec7da8 0000000000000082 ffff880037ec7d98 ffffffff8109bb05 ffff880000000000 000000018109ac0b ffff880037ec7d48 ffffffff8109b1c1 ffff880037c17058 ffff880037ec7fd8 000000000000fb88 ffff880037c17058 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? lock_hrtimer_base+0x31/0x60 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] ep_poll+0x2ad/0x330 [] ? default_wake_function+0x0/0x20 [] sys_epoll_wait+0xc5/0xe0 [] system_call_fastpath+0x16/0x1b sssd_nss S 0000000000000000 0 1268 1266 0x00000080 ffff88007b8cfda8 0000000000000086 ffff88007b8cfd98 ffffffff8109bb05 ffff880000000000 000000017b8cfd08 ffff88007b8cfd48 ffff88007b8cfd88 ffff88007dfc4638 ffff88007b8cffd8 000000000000fb88 ffff88007dfc4638 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] ep_poll+0x2ad/0x330 [] ? default_wake_function+0x0/0x20 [] sys_epoll_wait+0xc5/0xe0 [] system_call_fastpath+0x16/0x1b sssd_pam S 0000000000000000 0 1269 1266 0x00000080 ffff88007cc9fda8 0000000000000082 ffff88007cc9fd98 ffffffff8109bb05 ffff880000000000 000000018109ac0b ffff88007cc9fd48 ffffffff8109b1c1 ffff880037889058 ffff88007cc9ffd8 000000000000fb88 ffff880037889058 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? lock_hrtimer_base+0x31/0x60 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] ep_poll+0x2ad/0x330 [] ? default_wake_function+0x0/0x20 [] sys_epoll_wait+0xc5/0xe0 [] system_call_fastpath+0x16/0x1b rpc.statd S 0000000000000000 0 1287 1 0x00000080 ffff88003784d908 0000000000000082 ffff880002219c80 0000000000000034 ffff88003784d898 ffffffff813a16a5 0000000000000000 0000000000000042 ffff880037889ab8 ffff88003784dfd8 000000000000fb88 ffff880037889ab8 Call Trace: [] ? loopback_xmit+0x75/0xa0 [] ? ip_generic_getfrag+0x0/0xb0 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? udp_send_skb+0x119/0x3d0 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? handle_pte_fault+0x2cd/0xb50 [] ? move_addr_to_kernel+0x64/0x70 [] core_sys_select+0x18a/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? __d_free+0x3f/0x60 [] ? d_free+0x58/0x60 [] ? mntput_no_expire+0x30/0x110 [] sys_select+0x47/0x110 [] ? audit_syscall_entry+0x1d7/0x200 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b rpciod/0 S 0000000000000000 0 1315 2 0x00000080 ffff88007d8c9e30 0000000000000046 ffff88007d8c9d90 ffffffff810913cf ffff88007d8c9da0 ffffffffa025c06b ffff88007d8c9e10 ffffffffa025cabf ffff880037c3fab8 ffff88007d8c9fd8 000000000000fb88 ffff880037c3fab8 Call Trace: [] ? queue_work+0x1f/0x30 [] ? rpc_final_put_task+0x4b/0x60 [sunrpc] [] ? __rpc_execute+0x19f/0x350 [sunrpc] [] ? prepare_to_wait+0x4e/0x80 [] ? rpc_async_schedule+0x0/0x40 [sunrpc] [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 dbus-daemon S 0000000000000000 0 1413 1 0x00000080 ffff880037f69a58 0000000000000082 ffff880002216740 ffff880002216740 ffff880002216740 0000000000000001 0000000000000000 0000000000000000 ffff880037edbaf8 ffff880037f69fd8 000000000000fb88 ffff880037edbaf8 Call Trace: [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? do_sync_readv_writev+0xfb/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? do_readv_writev+0x169/0x1f0 [] ? __d_free+0x3f/0x60 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b kjournald S 0000000000000000 0 1447 2 0x00000080 ffff880037929e60 0000000000000046 ffff880037929dd0 ffffffff81096cb6 ffff880037828ad8 000000007a17fc80 ffff880037929e20 ffff880037da8000 ffff880037ffb098 ffff880037929fd8 000000000000fb88 ffff880037ffb098 Call Trace: [] ? autoremove_wake_function+0x16/0x40 [] ? prepare_to_wait+0x4e/0x80 [] kjournald+0x22a/0x250 [jbd] [] ? autoremove_wake_function+0x0/0x40 [] ? kjournald+0x0/0x250 [jbd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 acpid S 0000000000000000 0 1455 1 0x00000080 ffff88007b365a58 0000000000000086 ffff880037f4f500 0000000000000000 ffff88007b3659f8 ffffffff81065905 ffff880002216740 ffff880037b11578 ffff880037b11af8 ffff88007b365fd8 000000000000fb88 ffff880037b11af8 Call Trace: [] ? enqueue_entity+0x125/0x410 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __wake_up_common+0x59/0x90 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? sock_sendmsg+0x123/0x150 [] ? __put_single_page+0x1e/0x30 [] ? autoremove_wake_function+0x0/0x40 [] ? unix_write_space+0x1d/0x80 [] ? __wake_up+0x53/0x70 [] ? sys_sendto+0x139/0x190 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? getnstimeofday+0x60/0xf0 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b hald S 0000000000000000 0 1464 1 0x00000080 ffff88007db53a58 0000000000000086 ffff88007db53a48 ffffffff8109bb05 0000001000000000 00000001000200d0 ffff88000002bb08 0000000000000000 ffff880037b43ab8 ffff88007db53fd8 000000000000fb88 ffff880037b43ab8 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] ? poll_select_set_timeout+0x8d/0xa0 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b hald-runner S 0000000000000000 0 1465 1464 0x00000080 ffff88007d5d3a58 0000000000000082 ffff880002216740 ffff880002216740 ffff880002216740 0000000000000001 0000000000000000 0000000000000000 ffff880037b10638 ffff88007d5d3fd8 000000000000fb88 ffff880037b10638 Call Trace: [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] ? __wake_up_common+0x59/0x90 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? do_wp_page+0x493/0x920 [] ? handle_pte_fault+0x2cd/0xb50 [] ? autoremove_wake_function+0x0/0x40 [] ? sock_aio_write+0x0/0x1c0 [] ? do_sync_readv_writev+0xfb/0x140 [] ? handle_mm_fault+0x23a/0x310 [] ? wait_consider_task+0x9d/0xb20 [] ? remove_wait_queue+0x3c/0x50 [] ? do_wait+0x17f/0x240 [] ? sys_wait4+0xae/0x100 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b hald-addon-in S 0000000000000000 0 1493 1465 0x00000080 ffff88007d209a58 0000000000000086 ffff88007d209a68 ffff8800022115e8 ffff88007d2099f8 ffffffff8109ac0b ffff88007d2099f8 ffff88007d2099f8 ffff88007d040638 ffff88007d209fd8 000000000000fb88 ffff88007d040638 Call Trace: [] ? __remove_hrtimer+0x3b/0xb0 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? sock_aio_write+0x0/0x1c0 [] ? do_sync_readv_writev+0xfb/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? do_filp_open+0x72a/0xdd0 [] ? do_readv_writev+0x169/0x1f0 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b hald-addon-ac S 0000000000000000 0 1508 1465 0x00000080 ffff880037925b08 0000000000000086 ffff880037a193c0 00007f3f196d3000 ffff88007ccae850 0000000000000002 ffff880037925b98 ffffffff8104757c ffff88007d5e7058 ffff880037925fd8 000000000000fb88 ffff88007d5e7058 Call Trace: [] ? __do_page_fault+0x1ec/0x480 [] schedule_timeout+0x215/0x2e0 [] unix_stream_recvmsg+0x4c1/0x730 [] ? autoremove_wake_function+0x0/0x40 [] ? try_to_wake_up+0x24e/0x3e0 [] sock_aio_read+0x1a1/0x1b0 [] ? __vm_enough_memory+0x3b/0x190 [] do_sync_read+0xfa/0x140 [] ? autoremove_wake_function+0x0/0x40 [] ? do_mmap_pgoff+0x33a/0x380 [] ? security_file_permission+0x16/0x20 [] vfs_read+0x181/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b automount S 0000000000000000 0 1523 1 0x00000080 ffff88007cd0bdc8 0000000000000086 0000000000000004 ffff88007cb5d5a0 ffff880037975c40 ffff88007cb5d4e8 00000001ffffffea ffff880037828040 ffff8800378285f8 ffff88007cd0bfd8 000000000000fb88 ffff8800378285f8 Call Trace: [] ? enqueue_entity+0x125/0x410 [] schedule_timeout+0x215/0x2e0 [] ? __wake_up_sync_key+0x5e/0x80 [] schedule_timeout_interruptible+0x1e/0x20 [] sys_rt_sigtimedwait+0x206/0x260 [] ? __fput+0x1a1/0x210 [] ? audit_syscall_entry+0x1d7/0x200 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b automount S 0000000000000000 0 1524 1 0x00000080 ffff88007cdcdbf8 0000000000000086 ffff88007cdcdbe8 ffffffff8109bb05 ffffffff00000000 000000017cdcdc38 0000000000000000 00007f8f70ebce70 ffff880037829058 ffff88007cdcdfd8 000000000000fb88 ffff880037829058 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] futex_wait_queue_me+0xb9/0xf0 [] futex_wait+0x1f8/0x380 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] do_futex+0x121/0xb60 [] ? pvclock_clocksource_read+0x58/0xd0 [] sys_futex+0x7b/0x170 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b automount S 0000000000000000 0 1525 1 0x00000080 ffff88007b361bf8 0000000000000086 0000000000000246 ffffffff8112bbc3 01ff8800000126c0 0000000000000001 0000000000000000 0000000000000000 ffff88007d5e65f8 ffff88007b361fd8 000000000000fb88 ffff88007d5e65f8 Call Trace: [] ? __alloc_pages_nodemask+0x113/0x8d0 [] futex_wait_queue_me+0xb9/0xf0 [] futex_wait+0x1f8/0x380 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? futex_wake+0x10e/0x120 [] ? enqueue_entity+0x125/0x410 [] do_futex+0x121/0xb60 [] ? enqueue_task_fair+0xfb/0x100 [] ? check_preempt_curr+0x7c/0x90 [] ? wake_up_new_task+0xd9/0x130 [] ? do_fork+0x133/0x460 [] sys_futex+0x7b/0x170 [] ? audit_syscall_entry+0x1d7/0x200 [] ? __audit_syscall_exit+0x265/0x290 [] ? stub_clone+0x13/0x20 [] system_call_fastpath+0x16/0x1b automount S 0000000000000000 0 1528 1 0x00000080 ffff88007ccbfa58 0000000000000086 0000000000000282 0000000000000020 ffff88007cb6f118 ffff8800000126c0 ffff88007ccbf9e8 0000000000000001 ffff8800379abaf8 ffff88007ccbffd8 000000000000fb88 ffff8800379abaf8 Call Trace: [] ? inotify_d_instantiate+0x2a/0x60 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? check_preempt_wakeup+0x1a4/0x260 [] ? enqueue_task_fair+0x64/0x100 [] ? check_preempt_curr+0x7c/0x90 [] ? try_to_wake_up+0x24e/0x3e0 [] ? wake_up_state+0x10/0x20 [] ? wake_futex+0x40/0x60 [] ? do_futex+0x6b2/0xb60 [] ? vfs_ioctl+0x22/0xa0 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? sys_futex+0x7b/0x170 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b automount S 0000000000000000 0 1531 1 0x00000080 ffff88007cd59a58 0000000000000086 ffff88007cd599c8 ffffffffa0086eca ffff88007cb72a78 ffff88007cb64748 ffff88007cd599e8 ffffffff81096b67 ffff88007d041af8 ffff88007cd59fd8 000000000000fb88 ffff88007d041af8 Call Trace: [] ? wait_on_buffer+0x2a/0x40 [ext3] [] ? bit_waitqueue+0x17/0xd0 [] ? unlock_new_inode+0x4a/0x70 [] ? inotify_d_instantiate+0x2a/0x60 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? check_preempt_curr+0x7c/0x90 [] ? try_to_wake_up+0x24e/0x3e0 [] ? wake_up_state+0x10/0x20 [] ? wake_futex+0x40/0x60 [] ? futex_wake+0x10e/0x120 [] ? perf_event_task_sched_out+0x33/0x80 [] ? __dequeue_entity+0x30/0x50 [] ? __switch_to+0x1ac/0x320 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? getnstimeofday+0x60/0xf0 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b automount S 0000000000000000 0 1532 1 0x00000080 ffff88007cd57a58 0000000000000086 ffff880000000001 0000000000000001 0000000000000002 ffff880000034c20 ffff88007d040ae0 00000000000c12d0 ffff88007d041098 ffff88007cd57fd8 000000000000fb88 ffff88007d041098 Call Trace: [] ? unlock_new_inode+0x4a/0x70 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? check_preempt_curr+0x7c/0x90 [] ? try_to_wake_up+0x24e/0x3e0 [] ? wake_up_state+0x10/0x20 [] ? wake_futex+0x40/0x60 [] ? futex_wake+0x10e/0x120 [] ? perf_event_task_sched_out+0x33/0x80 [] ? __dequeue_entity+0x30/0x50 [] ? __switch_to+0x26e/0x320 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? getnstimeofday+0x60/0xf0 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b automount S 0000000000000000 0 1533 1 0x00000080 ffff88007cd45a58 0000000000000086 ffff88007cd459c8 ffffffffa0086eca ffff88007cb75d98 ffff88007cb64678 ffff88007cd459e8 ffffffff81096b67 ffff880037b425f8 ffff88007cd45fd8 000000000000fb88 ffff880037b425f8 Call Trace: [] ? wait_on_buffer+0x2a/0x40 [ext3] [] ? bit_waitqueue+0x17/0xd0 [] ? unlock_new_inode+0x4a/0x70 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? sched_clock_local+0x25/0x90 [] ? futex_wake+0x10e/0x120 [] ? enqueue_entity+0x125/0x410 [] ? do_futex+0x100/0xb60 [] ? enqueue_task_fair+0xfb/0x100 [] ? wake_up_new_task+0xd9/0x130 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? sys_futex+0x7b/0x170 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b rpc.rquotad S 0000000000000000 0 1565 1 0x00000080 ffff88007ce5da58 0000000000000082 ffff88007cd5b3c0 ffff88007ce5da68 ffff88007ce5da28 ffffffff8111b0c3 c900000000f5048b ffff88007ca44238 ffff88003799f058 ffff88007ce5dfd8 000000000000fb88 ffff88003799f058 Call Trace: [] ? filemap_fault+0xd3/0x500 [] ? unlock_page+0x27/0x30 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? handle_pte_fault+0xf7/0xb50 [] ? pte_alloc_one+0x37/0x50 [] ? __dentry_open+0x23f/0x360 [] ? handle_mm_fault+0x23a/0x310 [] ? cp_new_stat+0xe4/0x100 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b rpc.mountd S 0000000000000000 0 1569 1 0x00000080 ffff88007ce41908 0000000000000086 000000105b04468d 00000000000280da 000000100002bb08 00000000000200d0 ffff88000002bb08 0000000000000000 ffff88007a8c7ab8 ffff88007ce41fd8 000000000000fb88 ffff88007a8c7ab8 Call Trace: [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? check_preempt_wakeup+0x1c0/0x260 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] core_sys_select+0x18a/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? __switch_to+0x1ac/0x320 [] sys_select+0x47/0x110 [] ? audit_syscall_entry+0x1d7/0x200 [] system_call_fastpath+0x16/0x1b rpc.idmapd S 0000000000000000 0 1618 1 0x00000080 ffff88007b347da8 0000000000000086 ffff88007ba64aa0 ffffffff81096ca0 ffff88007b347d08 ffff88007b347d08 ffff88007b347d68 ffff88007b347d88 ffff88007ba65058 ffff88007b347fd8 000000000000fb88 ffff88007ba65058 Call Trace: [] ? autoremove_wake_function+0x0/0x40 [] schedule_hrtimeout_range+0x13d/0x160 [] ? ep_scan_ready_list+0x194/0x1a0 [] ep_poll+0x2ad/0x330 [] ? default_wake_function+0x0/0x20 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] sys_epoll_wait+0xc5/0xe0 [] system_call_fastpath+0x16/0x1b lockd S 0000000000000000 0 1622 2 0x00000080 ffff88007b341d10 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff88007d5e7ab8 ffff88007b341fd8 000000000000fb88 ffff88007d5e7ab8 Call Trace: [] ? pvclock_clocksource_read+0x58/0xd0 [] schedule_timeout+0x215/0x2e0 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] ? queue_delayed_work+0x21/0x40 [] lockd+0xc1/0x230 [lockd] [] ? lockd+0x0/0x230 [lockd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd4 S 0000000000000000 0 1623 2 0x00000080 ffff88007cecbe30 0000000000000046 ffff88007cecbdc0 ffffffff81091a89 0000000051a1ae91 0000000000000000 ffff88007cecbdc0 0000000000015f90 ffff88007ba645f8 ffff88007cecbfd8 000000000000fb88 ffff88007ba645f8 Call Trace: [] ? queue_delayed_work_on+0xb9/0x120 [] ? prepare_to_wait+0x4e/0x80 [] ? laundromat_main+0x0/0x3f0 [nfsd] [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd4_callbac S 0000000000000000 0 1624 2 0x00000080 ffff88007b77be30 0000000000000046 ffff88003799f548 ffff8800022167a8 ffff88007b77bdb0 ffffffff810572f0 ffff88003799f538 ffff8800022167a8 ffff88007ba6b098 ffff88007b77bfd8 000000000000fb88 ffff88007ba6b098 Call Trace: [] ? __dequeue_entity+0x30/0x50 [] ? prepare_to_wait+0x4e/0x80 [] ? worker_thread+0x0/0x2a0 [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd S 0000000000000000 0 1625 2 0x00000080 ffff88007ce99d70 0000000000000046 ffff88007ce99d80 ffffffffa026dd1b ffff88007ce99d30 0000000000000282 ffff88007ce99d10 ffffffff81080fcc ffff88007a8c7058 ffff88007ce99fd8 000000000000fb88 ffff88007a8c7058 Call Trace: [] ? svc_xprt_enqueue+0x12b/0x220 [sunrpc] [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] ? wake_up_process+0x15/0x20 [] nfsd+0xa5/0x160 [nfsd] [] ? nfsd+0x0/0x160 [nfsd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd S 0000000000000000 0 1626 2 0x00000080 ffff880037fbbd70 0000000000000046 ffff88007ba6b588 0000000000000286 ffff880037fbbd20 0000000000000282 ffff880037fbbd10 ffffffff81080fcc ffff88007ba6a638 ffff880037fbbfd8 000000000000fb88 ffff88007ba6a638 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] ? wake_up_process+0x15/0x20 [] nfsd+0xa5/0x160 [nfsd] [] ? nfsd+0x0/0x160 [nfsd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd S 0000000000000000 0 1627 2 0x00000080 ffff88007b23dd70 0000000000000046 ffff880037ac7548 0000000000000282 ffff88007b23dd20 0000000000000282 ffff88007b23dd10 ffffffff81080fcc ffff88007ba6baf8 ffff88007b23dfd8 000000000000fb88 ffff88007ba6baf8 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] ? wake_up_process+0x15/0x20 [] nfsd+0xa5/0x160 [nfsd] [] ? nfsd+0x0/0x160 [nfsd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd S 0000000000000000 0 1628 2 0x00000080 ffff88007b81fd70 0000000000000046 ffff880037ac6ae8 ffff8800022167a8 ffff88007b81fcf0 0000000000000282 ffff88007b81fd10 ffffffff81080fcc ffff880037ac7ab8 ffff88007b81ffd8 000000000000fb88 ffff880037ac7ab8 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] ? wake_up_process+0x15/0x20 [] nfsd+0xa5/0x160 [nfsd] [] ? nfsd+0x0/0x160 [nfsd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd S 0000000000000000 0 1629 2 0x00000080 ffff88007d541d70 0000000000000046 ffff88007d923548 ffff8800022167a8 ffff88007d541cf0 0000000000000282 ffff88007d541d10 ffffffff81080fcc ffff880037ac7058 ffff88007d541fd8 000000000000fb88 ffff880037ac7058 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] ? wake_up_process+0x15/0x20 [] nfsd+0xa5/0x160 [nfsd] [] ? nfsd+0x0/0x160 [nfsd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd S 0000000000000000 0 1630 2 0x00000080 ffff88007d925d70 0000000000000046 ffff88007d922ae8 ffff8800022167a8 ffff88007d925cf0 0000000000000282 ffff88007d925d10 ffffffff81080fcc ffff88007d923ab8 ffff88007d925fd8 000000000000fb88 ffff88007d923ab8 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] ? wake_up_process+0x15/0x20 [] nfsd+0xa5/0x160 [nfsd] [] ? nfsd+0x0/0x160 [nfsd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd S 0000000000000000 0 1631 2 0x00000080 ffff88007d887d70 0000000000000046 ffff88007d922088 ffff8800022167a8 ffff88007d887cf0 0000000000000282 ffff88007d887d10 ffffffff81080fcc ffff88007d923058 ffff88007d887fd8 000000000000fb88 ffff88007d923058 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] nfsd+0xa5/0x160 [nfsd] [] ? nfsd+0x0/0x160 [nfsd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsd S 0000000000000000 0 1632 2 0x00000080 ffff88007d94bd70 0000000000000046 ffff88007d94bd10 ffffffff8103c7b8 ffff880037aec0c0 0000000000000282 ffff88007d94bd10 ffffffff81080fcc ffff88007d9225f8 ffff88007d94bfd8 000000000000fb88 ffff88007d9225f8 Call Trace: [] ? pvclock_clocksource_read+0x58/0xd0 [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] svc_recv+0x5a5/0x850 [sunrpc] [] ? default_wake_function+0x0/0x20 [] nfsd+0xa5/0x160 [nfsd] [] ? nfsd+0x0/0x160 [nfsd] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 munged S 0000000000000000 0 1661 1 0x00000080 ffff88007d9a7c98 0000000000000082 0000000051a1ae91 ffffffff81075787 0000000000000036 ffff88007cbb3238 ffff88007d9a7e48 00000000000006e3 ffff88007ce85ab8 ffff88007d9a7fd8 000000000000fb88 ffff88007ce85ab8 Call Trace: [] ? current_fs_time+0x27/0x30 [] schedule_timeout+0x215/0x2e0 [] ? __generic_file_aio_write+0x260/0x490 [] ? pvclock_clocksource_read+0x58/0xd0 [] __skb_recv_datagram+0x237/0x290 [] ? receiver_wake_function+0x0/0x20 [] skb_recv_datagram+0x24/0x30 [] unix_accept+0x74/0x130 [] sys_accept4+0x155/0x2b0 [] ? vfs_write+0x132/0x1a0 [] ? audit_syscall_entry+0x1d7/0x200 [] sys_accept+0x10/0x20 [] system_call_fastpath+0x16/0x1b munged S 0000000000000000 0 1662 1 0x00000080 ffff88007db07bf8 0000000000000082 ffff88007db07be8 ffffffff8109bb05 0000000000000000 00000001019c43a8 ffff88007db07bd8 ffffffffa008b690 ffff88007ce0c638 ffff88007db07fd8 000000000000fb88 ffff88007ce0c638 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? ext3_ordered_write_end+0x100/0x1c0 [ext3] [] futex_wait_queue_me+0xb9/0xf0 [] futex_wait+0x1f8/0x380 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] do_futex+0x121/0xb60 [] ? user_path_at+0x62/0xa0 [] ? _atomic_dec_and_lock+0x55/0x80 [] ? cp_new_stat+0xe4/0x100 [] ? pvclock_clocksource_read+0x58/0xd0 [] sys_futex+0x7b/0x170 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b munged S 0000000000000000 0 1664 1 0x00000080 ffff88007cdd7bf8 0000000000000082 ffff88007cdd7b58 ffffffff8127f3ec ffff88007cdd7c28 ffffffff8127f6de ffff88007f25c140 0000000000001000 ffff88007ba65ab8 ffff88007cdd7fd8 000000000000fb88 ffff88007ba65ab8 Call Trace: [] ? put_dec+0x10c/0x110 [] ? number+0x2ee/0x320 [] futex_wait_queue_me+0xb9/0xf0 [] futex_wait+0x1f8/0x380 [] ? perf_event_task_sched_out+0x33/0x80 [] ? dequeue_entity+0x113/0x2e0 [] ? __dequeue_entity+0x30/0x50 [] do_futex+0x121/0xb60 [] ? get_current_tty+0x66/0x70 [] ? switch_task_namespaces+0x24/0x60 [] sys_futex+0x7b/0x170 [] ? audit_syscall_entry+0x1d7/0x200 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b munged S 0000000000000000 0 1665 1 0x00000080 ffff88007d945bf8 0000000000000082 000000000001a85d 0000000000000000 ffff88007d945b98 ffff8800022167a8 ffff8800022167d8 ffff8800022167a8 ffff88003799fab8 ffff88007d945fd8 000000000000fb88 ffff88003799fab8 Call Trace: [] futex_wait_queue_me+0xb9/0xf0 [] futex_wait+0x1f8/0x380 [] ? perf_event_task_sched_out+0x33/0x80 [] do_futex+0x121/0xb60 [] ? __do_page_fault+0x1ec/0x480 [] ? switch_task_namespaces+0x24/0x60 [] sys_futex+0x7b/0x170 [] ? audit_syscall_entry+0x1d7/0x200 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b sshd S 0000000000000000 0 1676 1 0x00000080 ffff88007d941908 0000000000000082 ffff88007eb883a8 0000000000000001 0000000100000000 ffff88007cbbed98 ffff88007cbbecc8 00000000003fed82 ffff88007ce0daf8 ffff88007d941fd8 000000000000fb88 ffff88007ce0daf8 Call Trace: [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? ext3_ordered_write_end+0x100/0x1c0 [ext3] [] ? find_get_page+0x1e/0xa0 [] ? filemap_fault+0xd3/0x500 [] ? unlock_page+0x27/0x30 [] ? __do_fault+0x469/0x530 [] ? handle_pte_fault+0xf7/0xb50 [] ? free_hot_page+0x2f/0x60 [] ? __put_single_page+0x1e/0x30 [] ? put_page+0x25/0x40 [] ? free_page_and_swap_cache+0x2a/0x60 [] core_sys_select+0x18a/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? cpumask_any_but+0x31/0x50 [] ? unmap_region+0xff/0x130 [] ? mntput_no_expire+0x30/0x110 [] sys_select+0x47/0x110 [] ? audit_syscall_entry+0x1d7/0x200 [] system_call_fastpath+0x16/0x1b xinetd S 0000000000000000 0 1684 1 0x00000080 ffff88007db1ba58 0000000000000086 0000000337f4f500 0000000000000000 ffff88000001a718 00000037ffffffc8 ffff88000002bb00 0000000000000000 ffff88007ce845f8 ffff88007db1bfd8 000000000000fb88 ffff88007ce845f8 Call Trace: [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_sys_poll+0x457/0x520 [] ? __wake_up_common+0x59/0x90 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? autoremove_wake_function+0x0/0x40 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? handle_mm_fault+0x23a/0x310 [] ? enqueue_task_fair+0xfb/0x100 [] ? sys_sendto+0x139/0x190 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? getnstimeofday+0x60/0xf0 [] sys_poll+0x71/0x100 [] system_call_fastpath+0x16/0x1b ntpd S 0000000000000000 0 1692 1 0x00000080 ffff88007d9bd908 0000000000000082 ffff88007d9bd8f8 ffffffff8109bb05 ffff880000000000 0000000100000001 ffff88007d9bc000 0000000000013200 ffff88007ce85058 ffff88007d9bdfd8 000000000000fb88 ffff88007ce85058 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? __generic_file_aio_write+0x260/0x490 [] core_sys_select+0x18a/0x2c0 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] sys_select+0x47/0x110 [] system_call_fastpath+0x16/0x1b sendmail S 0000000000000000 0 1708 1 0x00000080 ffff88007d505908 0000000000000082 ffff88007d5058f8 ffffffff8109bb05 ffff880000000000 0000000100000286 ffff88007d5058a8 ffffffff8109b1c1 ffff88007dbc1ab8 ffff88007d505fd8 000000000000fb88 ffff88007dbc1ab8 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] ? lock_hrtimer_base+0x31/0x60 [] ? get_page_from_freelist+0x3d1/0x830 [] schedule_hrtimeout_range+0xc8/0x160 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? apic_timer_interrupt+0xe/0x20 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? inode_init_always+0x11e/0x1c0 [] ? bit_waitqueue+0x17/0xd0 [] ? inotify_d_instantiate+0x2a/0x60 [] ? __d_instantiate+0xbb/0x110 [] ? put_dec+0x10c/0x110 [] ? number+0x2ee/0x320 [] ? proc_lookup+0x1b/0x20 [] ? vsnprintf+0x336/0x5e0 [] ? single_open+0x6f/0xa0 [] ? seq_vprintf+0x32/0x60 [] core_sys_select+0x18a/0x2c0 [] ? fsnotify_clear_marks_by_inode+0x32/0xf0 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? kvm_clock_read+0x1c/0x20 [] ? kvm_clock_get_cycles+0x9/0x10 [] ? ktime_get_ts+0xa9/0xe0 [] sys_select+0x47/0x110 [] system_call_fastpath+0x16/0x1b sendmail S 0000000000000000 0 1716 1 0x00000080 ffff88007d53df68 0000000000000086 ffff88007dbc9a48 ffff88007dbc9a48 ffff88007dbc9a48 ffff88007d53df10 ffffffff810dc645 00007ffff5a960f8 ffff88007ce0d098 ffff88007d53dfd8 000000000000fb88 ffff88007ce0d098 Call Trace: [] ? __audit_syscall_exit+0x265/0x290 [] ? audit_syscall_entry+0x1d7/0x200 [] sys_pause+0x1e/0x30 [] system_call_fastpath+0x16/0x1b crond S 0000000000000000 0 1738 1 0x00000080 ffff88007db31e68 0000000000000082 ffff88007db31e58 ffffffff8109bb05 ffff880000000000 000000018107196f 00000000002cdc86 00007fffffffeffd ffff8800379ab098 ffff88007db31fd8 000000000000fb88 ffff8800379ab098 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] do_nanosleep+0x8b/0xc0 [] hrtimer_nanosleep+0xc4/0x180 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] sys_nanosleep+0x74/0x80 [] system_call_fastpath+0x16/0x1b atd S 0000000000000000 0 1749 1 0x00000080 ffff88007db19e68 0000000000000086 ffff88007db19e58 ffffffff8109bb05 ffff880000000000 00000001811926f2 ffff8800797868c0 ffff88007cbe8b00 ffff88007db1d098 ffff88007db19fd8 000000000000fb88 ffff88007db1d098 Call Trace: [] ? __hrtimer_start_range_ns+0x1a5/0x470 [] do_nanosleep+0x8b/0xc0 [] hrtimer_nanosleep+0xc4/0x180 [] ? hrtimer_wakeup+0x0/0x30 [] ? hrtimer_start_range_ns+0x14/0x20 [] sys_nanosleep+0x74/0x80 [] system_call_fastpath+0x16/0x1b mingetty S 0000000000000000 0 1775 1 0x00000080 ffff880037a6fcc8 0000000000000086 ffff880037a6fca8 ffffffff811435a9 00000000023e0025 0000000000000000 0000000000000000 ffffea0001b6d358 ffff880037c3f058 ffff880037a6ffd8 000000000000fb88 ffff880037c3f058 Call Trace: [] ? __do_fault+0x469/0x530 [] schedule_timeout+0x215/0x2e0 [] ? flush_work+0x21/0xc0 [] ? try_to_del_timer_sync+0x7b/0xe0 [] n_tty_read+0x5f7/0x990 [] ? release_console_sem+0x1cf/0x220 [] ? default_wake_function+0x0/0x20 [] tty_read+0xa6/0xf0 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b mingetty S 0000000000000000 0 1777 1 0x00000080 ffff88007dbc3cc8 0000000000000082 ffff88007dbc3ca8 ffffffff811435a9 00000000023e0025 0000000000000000 0000000000000000 ffffea0001b6cda8 ffff88007dbc1058 ffff88007dbc3fd8 000000000000fb88 ffff88007dbc1058 Call Trace: [] ? __do_fault+0x469/0x530 [] schedule_timeout+0x215/0x2e0 [] ? flush_work+0x21/0xc0 [] ? try_to_del_timer_sync+0x7b/0xe0 [] n_tty_read+0x5f7/0x990 [] ? release_console_sem+0x1cf/0x220 [] ? default_wake_function+0x0/0x20 [] tty_read+0xa6/0xf0 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b mingetty S 0000000000000000 0 1779 1 0x00000080 ffff88007dbdbcc8 0000000000000086 ffff88007dbdbca8 ffffffff811435a9 00000000023e0025 0000000000000000 0000000000000000 ffffea0001b6ce50 ffff88003792a638 ffff88007dbdbfd8 000000000000fb88 ffff88003792a638 Call Trace: [] ? __do_fault+0x469/0x530 [] schedule_timeout+0x215/0x2e0 [] ? flush_work+0x21/0xc0 [] ? try_to_del_timer_sync+0x7b/0xe0 [] n_tty_read+0x5f7/0x990 [] ? release_console_sem+0x1cf/0x220 [] ? default_wake_function+0x0/0x20 [] tty_read+0xa6/0xf0 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b mingetty S 0000000000000000 0 1781 1 0x00000080 ffff88007d82bcc8 0000000000000082 ffff88007d82bca8 ffffffff811435a9 00000000023e0025 0000000000000000 0000000000000000 ffffea0001b6b058 ffff880037eda638 ffff88007d82bfd8 000000000000fb88 ffff880037eda638 Call Trace: [] ? __do_fault+0x469/0x530 [] schedule_timeout+0x215/0x2e0 [] ? flush_work+0x21/0xc0 [] ? try_to_del_timer_sync+0x7b/0xe0 [] n_tty_read+0x5f7/0x990 [] ? release_console_sem+0x1cf/0x220 [] ? default_wake_function+0x0/0x20 [] tty_read+0xa6/0xf0 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b mingetty S 0000000000000000 0 1783 1 0x00000080 ffff88007db81cc8 0000000000000082 ffff88007db81ca8 ffffffff811435a9 00000000023e0025 0000000000000000 0000000000000000 ffffea0001b73fd8 ffff88007dbc05f8 ffff88007db81fd8 000000000000fb88 ffff88007dbc05f8 Call Trace: [] ? __do_fault+0x469/0x530 [] schedule_timeout+0x215/0x2e0 [] ? flush_work+0x21/0xc0 [] ? try_to_del_timer_sync+0x7b/0xe0 [] n_tty_read+0x5f7/0x990 [] ? release_console_sem+0x1cf/0x220 [] ? default_wake_function+0x0/0x20 [] tty_read+0xa6/0xf0 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b mingetty S 0000000000000000 0 1785 1 0x00000080 ffff880037a65cc8 0000000000000082 ffff880037a65ca8 ffffffff811435a9 00000000023e0025 0000000000000000 0000000000000000 ffffea0001b7f300 ffff88003792baf8 ffff880037a65fd8 000000000000fb88 ffff88003792baf8 Call Trace: [] ? __do_fault+0x469/0x530 [] schedule_timeout+0x215/0x2e0 [] ? flush_work+0x21/0xc0 [] ? try_to_del_timer_sync+0x7b/0xe0 [] n_tty_read+0x5f7/0x990 [] ? release_console_sem+0x1cf/0x220 [] ? default_wake_function+0x0/0x20 [] tty_read+0xa6/0xf0 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b agetty S 0000000000000000 0 1786 1 0x00000080 ffff88007d7e9cc8 0000000000000086 ffff88007d7e9ca8 ffffffff811435a9 00000000023e0025 0000000000000000 0000000000000000 ffffea0001b8d920 ffff88007b37c638 ffff88007d7e9fd8 000000000000fb88 ffff88007b37c638 Call Trace: [] ? __do_fault+0x469/0x530 [] schedule_timeout+0x215/0x2e0 [] ? flush_work+0x21/0xc0 [] ? try_to_del_timer_sync+0x7b/0xe0 [] n_tty_read+0x5f7/0x990 [] ? uart_write+0xe9/0x120 [] ? default_wake_function+0x0/0x20 [] tty_read+0xa6/0xf0 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b in.rshd S 0000000000000000 0 2234 1684 0x00000080 ffff88007ceef908 0000000000000086 0000000000000000 0000000000000000 ffff88003744e020 ffff8800797f2da8 000000003744e020 ffff88007c01da40 ffff88003799e5f8 ffff88007ceeffd8 000000000000fb88 ffff88003799e5f8 Call Trace: [] ? get_page_from_freelist+0x3d1/0x830 [] schedule_hrtimeout_range+0x13d/0x160 [] ? add_wait_queue+0x46/0x60 [] ? __pollwait+0x75/0xf0 [] poll_schedule_timeout+0x39/0x60 [] do_select+0x57c/0x6c0 [] ? __pollwait+0x0/0xf0 [] ? pollwake+0x0/0x60 [] ? pollwake+0x0/0x60 [] ? __lru_cache_add+0x40/0x90 [] ? lru_cache_add_lru+0x21/0x40 [] ? page_add_new_anon_rmap+0x9d/0xf0 [] ? do_wp_page+0x35e/0x920 [] ? kmem_getpages+0xba/0x170 [] ? handle_pte_fault+0x2cd/0xb50 [] ? pvclock_clocksource_read+0x58/0xd0 [] core_sys_select+0x18a/0x2c0 [] ? __do_page_fault+0x1ec/0x480 [] ? wake_up_new_task+0xd9/0x130 [] ? do_fork+0x133/0x460 [] ? do_vfs_ioctl+0x2ff/0x580 [] sys_select+0x47/0x110 [] ? audit_syscall_entry+0x1d7/0x200 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b bash R running task 0 2235 2234 0x00000080 0000000000000000 ffffffff81ead300 0000000000000000 0000000000000000 ffff88007dbd9898 ffff88007dbcd2b0 ffffffff81ead540 000000000069fda9 3831303062303732 6666666666666666 000000000000fb88 ffff88007db1c638 Call Trace: [] ? pvclock_clocksource_read+0x58/0xd0 [] ? pvclock_clocksource_read+0x58/0xd0 [] ? pointer+0x22b/0x830 [] ? delay_tsc+0x4a/0x80 [] ? __const_udelay+0x46/0x50 [] ? wait_for_xmitr+0x62/0xe0 [] ? delay_tsc+0x4a/0x80 [] ? __const_udelay+0x46/0x50 [] ? wait_for_xmitr+0x62/0xe0 [] ? __call_console_drivers+0x75/0x90 [] ? up+0x2f/0x50 [] ? _call_console_drivers+0x4a/0x80 [] ? release_console_sem+0x1cf/0x220 [] ? __const_udelay+0x46/0x50 [] ? wait_for_xmitr+0x62/0xe0 [] ? wait_for_xmitr+0x62/0xe0 [] ? printk+0x41/0x48 [] ? printk+0x41/0x48 [] ? printk+0x41/0x48 [] ? __module_text_address+0x12/0x60 [] ? printk_address+0x31/0x40 [] ? is_module_text_address+0xe/0x20 [] ? __kernel_text_address+0x58/0x80 [] ? print_context_stack+0x85/0x140 [] ? dump_trace+0x190/0x3b0 [] show_trace_log_lvl+0x55/0x70 [] show_stack_log_lvl+0x9a/0x170 [] show_stack+0x1a/0x20 [] sched_show_task+0x98/0x100 [] show_state_filter+0x7b/0xc0 [] sysrq_handle_showstate+0x10/0x20 [] __handle_sysrq+0x132/0x1a0 [] write_sysrq_trigger+0x4e/0x50 [] proc_reg_write+0x7e/0xc0 [] vfs_write+0xb8/0x1a0 [] sys_write+0x51/0x90 [] system_call_fastpath+0x16/0x1b kslowd000 S 0000000000000000 0 2261 2 0x00000080 ffff88007d05be60 0000000000000046 000000010000000f 0000000a00000003 ffffffffffffffff ffffffff81065ceb ffff88007d05bdf0 ffff88007d05be80 ffff88007d1d85f8 ffff88007d05bfd8 000000000000fb88 ffff88007d1d85f8 Call Trace: [] ? enqueue_task_fair+0xfb/0x100 [] ? prepare_to_wait_exclusive+0x4e/0x80 [] slow_work_thread+0x32b/0x360 [] ? autoremove_wake_function+0x0/0x40 [] ? slow_work_thread+0x0/0x360 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 kslowd001 S 0000000000000000 0 2262 2 0x00000080 ffff88007db79e60 0000000000000046 000000010000000f 0000000a00000003 ffffffffffffffff ffffffff81065ceb ffff88007db79df0 ffff88007db79e80 ffff88007d05faf8 ffff88007db79fd8 000000000000fb88 ffff88007d05faf8 Call Trace: [] ? enqueue_task_fair+0xfb/0x100 [] ? prepare_to_wait_exclusive+0x4e/0x80 [] slow_work_thread+0x32b/0x360 [] ? autoremove_wake_function+0x0/0x40 [] ? slow_work_thread+0x0/0x360 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 nfsiod S 0000000000000000 0 2263 2 0x00000080 ffff880037fefe30 0000000000000046 ffff880037fee000 ffff880037fee000 ffff88007dbe9a80 0000000000000000 ffff88007dbe9a80 ffffffffa025c100 ffff88007d05f098 ffff880037feffd8 000000000000fb88 ffff88007d05f098 Call Trace: [] ? rpc_async_release+0x0/0x20 [sunrpc] [] ? prepare_to_wait+0x4e/0x80 [] ? rpc_async_release+0x0/0x20 [sunrpc] [] worker_thread+0x1fc/0x2a0 [] ? autoremove_wake_function+0x0/0x40 [] ? worker_thread+0x0/0x2a0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 Sched Debug Version: v0.09, 2.6.32-358.6.2.el6_lustre.g97b2c86.x86_64 #1 now at 7187220.042262 msecs .jiffies : 4301853513 .sysctl_sched_latency : 5.000000 .sysctl_sched_min_granularity : 1.000000 .sysctl_sched_wakeup_granularity : 1.000000 .sysctl_sched_child_runs_first : 0.000000 .sysctl_sched_features : 3183 .sysctl_sched_tunable_scaling : 1 (logaritmic) cpu#0, 2400.084 MHz .nr_running : 1 .load : 1024 .nr_switches : 81215 .nr_load_updates : 49278 .nr_uninterruptible : 0 .next_balance : 4301.854001 .curr->pid : 2235 .clock : 7186217.576785 .cpu_load[0] : 1024 .cpu_load[1] : 769 .cpu_load[2] : 473 .cpu_load[3] : 331 .cpu_load[4] : 313 .yld_count : 1 .sched_switch : 0 .sched_count : 81298 .sched_goidle : 27365 .avg_idle : 434092 .ttwu_count : 45702 .ttwu_local : 45702 .bkl_count : 166 cfs_rq[0]:/ .exec_clock : 6303.089393 .MIN_vruntime : 0.000001 .min_vruntime : 7089.823031 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_spread_over : 7 .nr_running : 1 .load : 1024 .load_avg : 0.000000 .load_period : 0.000000 .load_contrib : 0 .load_tg : 0 rt_rq[0]:/ .rt_nr_running : 0 .rt_throttled : 0 .rt_time : 0.000000 .rt_runtime : 950.000000 runnable tasks: task PID tree-key switches prio exec-runtime sum-exec sum-sleep ---------------------------------------------------------------------------------------------------------- R bash 2235 7089.823031 65 120 7089.823031 11.421642 1092.576451 /