Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Lustre 2.15.5
-
None
-
3
-
9223372036854775807
Description
The Lustre client has experienced memory exhaustion (total memory 375Gi, used 373Gi), leading to an interruption of the vdbench service.
1 Memory Info:
[root@client213 ~]# free -h
total used free shared buff/cache available
Mem: 375Gi 373Gi 2.9Gi 2.0Mi 1.6Gi 2.3Gi
Swap: 0B 0B 0B
cat /proc/slabinfo
...
kmalloc-64 6148242931 6148243648 64 64 1 : tunables 0 0 0 : slabdata 96066307 96066307 0
...
2 Memory Content Printed via crash:
0xff387f301e333140: 0x6e69666e6f636e75 0x6a626f3a755f6465
0xff387f301e333150: 0x6e753a725f746365 0x5f64656c6562616c
0xff387f301e333160: 0x0cd2ec0030733a74 0x5a5a5a5a5a5a5a5a
0xff387f301e333170: 0x5a5a5a5a5a5a5a5a 0x5a5a5a5a5a5a5a5a
3 Logs(/var/log/message)
Jun 23 18:12:22 client213 kernel: obd_memory max: 35736380442, obd_memory current: 108007944
Jun 23 18:12:22 client213 kernel: java invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jun 23 18:12:23 client213 kernel: CPU: 57 PID: 1111006 Comm: java Kdump: loaded Tainted: G OE 5.15.131-17.cl9.x86_64 #1
Jun 23 18:12:23 client213 kernel: Hardware name: XFUSION CeaLustre 5318Y/BC13MBSBC, BIOS 1.10 07/06/2022
Jun 23 18:12:25 client213 kernel: Call Trace:
Jun 23 18:12:25 client213 kernel: <TASK>
Jun 23 18:12:25 client213 kernel: dump_stack_lvl+0x34/0x48
Jun 23 18:12:25 client213 kernel: dump_header+0x4a/0x1f7
Jun 23 18:12:25 client213 kernel: oom_kill_process.cold+0xb/0x10
Jun 23 18:12:25 client213 kernel: out_of_memory+0xed/0x2d0
Jun 23 18:12:25 client213 kernel: __alloc_pages_slowpath.constprop.0+0x963/0xa20
Jun 23 18:12:25 client213 kernel: __alloc_pages+0x1e9/0x220
Jun 23 18:12:25 client213 kernel: pagecache_get_page+0x14c/0x520
Jun 23 18:12:25 client213 kernel: filemap_fault+0x46d/0xa60
Jun 23 18:12:25 client213 kernel: __do_fault+0x34/0x170
Jun 23 18:12:25 client213 kernel: do_fault+0x190/0x370
Jun 23 18:12:25 client213 kernel: __handle_mm_fault+0x3dd/0x700
Jun 23 18:12:25 client213 kernel: handle_mm_fault+0xbf/0x290
Jun 23 18:12:25 client213 kernel: do_user_addr_fault+0x1b7/0x670
Jun 23 18:12:25 client213 kernel: exc_page_fault+0x62/0x130
Jun 23 18:12:25 client213 kernel: asm_exc_page_fault+0x22/0x30
Jun 23 18:12:25 client213 kernel: RIP: 0033:0x7fcc3a734e4e
Jun 23 18:12:25 client213 kernel: Code: Unable to access opcode bytes at RIP 0x7fcc3a734e24.
Jun 23 18:12:25 client213 kernel: RSP: 002b:00007fcb5cdd0ce0 EFLAGS: 00010202
Jun 23 18:12:25 client213 kernel: RAX: 00000000003b3910 RBX: 00007fcc39797420 RCX: 0000000000000002
Jun 23 18:12:25 client213 kernel: RDX: 00007fcc3a08e000 RSI: 00007fcc3ada83cc RDI: 00007fcc34018a60
Jun 23 18:12:25 client213 kernel: RBP: 00007fcb5cdd0cf0 R08: 00007fcc3419e000 R09: 0000000000000032
Jun 23 18:12:25 client213 kernel: R10: 00007ffcfe5d4080 R11: 00007ffcfe5d4090 R12: 0000000000000008
Jun 23 18:12:25 client213 kernel: R13: 00007fcc3419dbc0 R14: 0000000000000002 R15: 0000000000000001
Jun 23 18:12:25 client213 kernel: </TASK>
Jun 23 18:12:25 client213 kernel: Mem-Info:
Jun 23 18:12:25 client213 kernel: active_anon:13078 inactive_anon:2044534 isolated_anon:0#012 active_file:0 inactive_file:9365 isolated_file:55#012 unevictable:0 dirty:537 writeback:329#012 slab_reclaimable:133180 slab_unreclaimable:95128779#012 mapped:3462 shmem:680 pagetables:10044 bounce:0#012 kernel_misc_reclaimable:0#012 free:451818 free_pcp:9408 free_cma:0
Jun 23 18:12:25 client213 kernel: Node 0 active_anon:29624kB inactive_anon:4694876kB active_file:0kB inactive_file:14784kB unevictable:0kB isolated(anon):0kB isolated(file):160kB mapped:7704kB dirty:2408kB writeback:1216kB shmem:2076kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:35724kB pagetables:27832kB all_unreclaimable? no
Jun 23 18:12:25 client213 kernel: Node 1 active_anon:22688kB inactive_anon:3483260kB active_file:0kB inactive_file:9232kB unevictable:0kB isolated(anon):0kB isolated(file):60kB mapped:3120kB dirty:0kB writeback:100kB shmem:644kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:28084kB pagetables:12344kB all_unreclaimable? no
Jun 23 18:12:25 client213 kernel: Node 0 DMA free:11264kB min:0kB low:12kB high:24kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 23 18:12:25 client213 kernel: lowmem_reserve[]: 0 928 192098 192098 192098
Jun 23 18:12:25 client213 kernel: Node 0 DMA32 free:764644kB min:188kB low:1136kB high:2084kB reserved_highatomic:0KB active_anon:0kB inactive_anon:85604kB active_file:160kB inactive_file:240kB unevictable:0kB writepending:740kB present:1281444kB managed:950600kB mlocked:0kB bounce:0kB free_pcp:10992kB local_pcp:0kB free_cma:0kB
Jun 23 18:12:25 client213 kernel: lowmem_reserve[]: 0 0 191170 191170 191170
Jun 23 18:12:25 client213 kernel: Node 0 Normal free:634176kB min:335680kB low:531436kB high:727192kB reserved_highatomic:2048KB active_anon:29624kB inactive_anon:4609280kB active_file:0kB inactive_file:8712kB unevictable:0kB writepending:3804kB present:198967296kB managed:195766828kB mlocked:0kB bounce:0kB free_pcp:18896kB local_pcp:0kB free_cma:0kB
Jun 23 18:12:25 client213 kernel: hrtimer: interrupt took 266799 ns
Jun 23 18:12:25 client213 kernel: lowmem_reserve[]: 0 0 0 0 0
Jun 23 18:12:25 client213 kernel: Node 1 Normal free:393544kB min:39740kB low:236940kB high:434140kB reserved_highatomic:2048KB active_anon:22688kB inactive_anon:3483260kB active_file:0kB inactive_file:13324kB unevictable:0kB writepending:96kB present:201326592kB managed:197236612kB mlocked:0kB bounce:0kB free_pcp:39636kB local_pcp:0kB free_cma:0kB
Jun 23 18:12:25 client213 kernel: lowmem_reserve[]: 0 0 0 0 0
Jun 23 18:12:25 client213 kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
Jun 23 18:12:25 client213 kernel: Node 0 DMA32: 954*4kB (UME) 1087*8kB (UME) 723*16kB (UME) 520*32kB (UME) 561*64kB (UME) 206*128kB (UME) 76*256kB (UME) 75*512kB (UME) 66*1024kB (UME) 26*2048kB (UME) 118*4096kB (UME) = 765008kB
Jun 23 18:12:25 client213 kernel: Node 0 Normal: 3106*4kB (UMEH) 5667*8kB (UMEH) 6213*16kB (UMEH) 15510*32kB (UMEH) 3*64kB (MH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 653680kB
Jun 23 18:12:25 client213 kernel: Node 1 Normal: 2793*4kB (UME) 2762*8kB (UME) 2085*16kB (UME) 11748*32kB (UMEH) 43*64kB (ME) 2*128kB (ME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 445572kB
Jun 23 18:12:25 client213 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 23 18:12:25 client213 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 23 18:12:25 client213 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 23 18:12:25 client213 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 23 18:12:25 client213 kernel: 12215 total pagecache pages
Jun 23 18:12:25 client213 kernel: 0 pages in swap cache
Jun 23 18:12:25 client213 kernel: Swap cache stats: add 0, delete 0, find 0/0
Jun 23 18:12:25 client213 kernel: Free swap = 0kB
Jun 23 18:12:25 client213 kernel: Total swap = 0kB
Jun 23 18:12:25 client213 kernel: 100397832 pages RAM
Jun 23 18:12:25 client213 kernel: 0 pages HighMem/MovableOnly
Jun 23 18:12:25 client213 kernel: 1905482 pages reserved
Jun 23 18:12:25 client213 kernel: 0 pages cma reserved
Jun 23 18:12:26 client213 kernel: 0 pages hwpoisoned
Jun 23 18:12:26 client213 kernel: Unreclaimable slab info:
Jun 23 18:12:27 client213 kernel: Name Used Total
Jun 23 18:12:27 client213 kernel: cl_page_kmem-320 46895KB 46907KB
Jun 23 18:12:27 client213 kernel: xattr_kmem 3601KB 3617KB
Jun 23 18:12:28 client213 kernel: vvp_thread_kmem 24946KB 24946KB
Jun 23 18:12:28 client213 kernel: vvp_session_kmem 21902KB 21906KB
Jun 23 18:12:28 client213 kernel: vvp_object_kmem 3778KB 4119KB
Jun 23 18:12:28 client213 kernel: ll_thread_kmem 10635KB 10682KB
Jun 23 18:12:28 client213 kernel: ll_file_data 2119KB 2119KB
Jun 23 18:12:28 client213 kernel: lov_oinfo 4173KB 4272KB
Jun 23 18:12:28 client213 kernel: lovsub_object_kmem 4191KB 4324KB
Jun 23 18:12:28 client213 kernel: lov_session_kmem 21047KB 21077KB
Jun 23 18:12:28 client213 kernel: lov_thread_kmem 11242KB 11280KB
Jun 23 18:12:28 client213 kernel: lov_object_kmem 4548KB 4703KB
Jun 23 18:12:28 client213 kernel: osc_obdo_kmem 11787KB 11787KB
Jun 23 18:12:28 client213 kernel: osc_extent_kmem 11292KB 11324KB
Jun 23 18:12:28 client213 kernel: osc_session_kmem 10853KB 10970KB
Jun 23 18:12:28 client213 kernel: osc_thread_kmem 15977KB 17545KB
Jun 23 18:12:28 client213 kernel: osc_object_kmem 3904KB 4382KB
Jun 23 18:12:28 client213 kernel: osc_lock_kmem 2016KB 2016KB
Jun 23 18:12:28 client213 kernel: interval_tree 10856KB 10912KB
Jun 23 18:12:28 client213 kernel: interval_node 9984KB 10000KB
Jun 23 18:12:28 client213 kernel: ldlm_locks 28081KB 28094KB
Jun 23 18:12:28 client213 kernel: ldlm_resources 9880KB 9918KB
Jun 23 18:12:28 client213 kernel: lnet_msg 9526KB 9526KB
Jun 23 18:12:28 client213 kernel: lnet_rspt 3KB 3KB
Jun 23 18:12:28 client213 kernel: lnet_small_MDs 9562KB 9562KB
Jun 23 18:12:28 client213 kernel: lnet_MEs 4012KB 4012KB
Jun 23 18:12:28 client213 kernel: ptlrpc_cache 14871KB 15108KB
Jun 23 18:12:28 client213 kernel: cl_sub_dio_kmem 1043KB 1043KB
Jun 23 18:12:28 client213 kernel: cl_dio_aio_kmem 545KB 545KB
Jun 23 18:12:28 client213 kernel: cl_env_kmem 11242KB 11265KB
Jun 23 18:12:28 client213 kernel: ll_obd_dev_cache 362KB 362KB
Jun 23 18:12:28 client213 kernel: rpc_buffers 32KB 32KB
Jun 23 18:12:28 client213 kernel: rpc_tasks 16KB 16KB
Jun 23 18:12:28 client213 kernel: ext4_system_zone 91KB 91KB
Jun 23 18:12:28 client213 kernel: ext4_io_end_vec 1884KB 1892KB
Jun 23 18:12:28 client213 kernel: ext4_bio_post_read_ctx 7KB 7KB
Jun 23 18:12:28 client213 kernel: jbd2_inode 6500KB 6532KB
Jun 23 18:12:28 client213 kernel: bio-168 15KB 15KB
Jun 23 18:12:28 client213 kernel: bio-232 22816KB 22896KB
Jun 23 18:12:28 client213 kernel: ioat 512KB 592KB
Jun 23 18:12:28 client213 kernel: xfs_efi_item 2029KB 2029KB
Jun 23 18:12:28 client213 kernel: xfs_efd_item 2013KB 2013KB
Jun 23 18:12:28 client213 kernel: xfs_buf_item 2046KB 2046KB
Jun 23 18:12:28 client213 kernel: xfs_trans 2030KB 2030KB
Jun 23 18:12:28 client213 kernel: xfs_ifork 1820KB 1840KB
Jun 23 18:12:28 client213 kernel: xfs_da_state 4080KB 4080KB
Jun 23 18:12:28 client213 kernel: xfs_btree_cur 1008KB 1008KB
Jun 23 18:12:28 client213 kernel: xfs_bmap_free_item 511KB 511KB
Jun 23 18:12:29 client213 kernel: xfs_log_ticket 1012KB 1012KB
Jun 23 18:12:29 client213 kernel: sd_ext_cdb 4KB 4KB
Jun 23 18:12:29 client213 kernel: fs_ftes_0000:98:00.1 63KB 63KB
Jun 23 18:12:29 client213 kernel: fs_fgs_0000:98:00.1 31KB 31KB
Jun 23 18:12:29 client213 kernel: fs_ftes_0000:98:00.0 63KB 63KB
Jun 23 18:12:29 client213 kernel: fs_fgs_0000:98:00.0 31KB 31KB
Jun 23 18:12:29 client213 kernel: fs_ftes_0000:31:00.1 63KB 63KB
Jun 23 18:12:29 client213 kernel: fs_fgs_0000:31:00.1 31KB 31KB
Jun 23 18:12:29 client213 kernel: fs_ftes_0000:31:00.0 158KB 158KB
Jun 23 18:12:29 client213 kernel: fs_fgs_0000:31:00.0 63KB 63KB
Jun 23 18:12:29 client213 kernel: scsi_sense_cache 28056KB 28056KB
Jun 23 18:12:29 client213 kernel: bio-120 14928KB 14928KB
Jun 23 18:12:29 client213 kernel: fib6_nodes 428KB 428KB
Jun 23 18:12:29 client213 kernel: ip6_dst_cache 2048KB 2048KB
Jun 23 18:12:29 client213 kernel: RAWv6 9324KB 9324KB
Jun 23 18:12:29 client213 kernel: UDPv6 4032KB 4032KB
Jun 23 18:12:29 client213 kernel: tw_sock_TCPv6 2046KB 2046KB
Jun 23 18:12:29 client213 kernel: TCPv6 3952KB 3952KB
Jun 23 18:12:29 client213 kernel: sgpool-128 32KB 32KB
Jun 23 18:12:29 client213 kernel: sgpool-64 64KB 64KB
Jun 23 18:12:29 client213 kernel: sgpool-32 4096KB 4096KB
Jun 23 18:12:29 client213 kernel: sgpool-16 4096KB 4096KB
Jun 23 18:12:29 client213 kernel: sgpool-8 2048KB 2048KB
Jun 23 18:12:29 client213 kernel: bio-208 112KB 112KB
Jun 23 18:12:29 client213 kernel: mqueue_inode_cache 31KB 31KB
Jun 23 18:12:29 client213 kernel: kioctx 189KB 189KB
Jun 23 18:12:29 client213 kernel: aio_kiocb 47KB 47KB
Jun 23 18:12:29 client213 kernel: audit_tree_mark 3KB 3KB
Jun 23 18:12:29 client213 kernel: posix_timers_cache 2048KB 2048KB
Jun 23 18:12:29 client213 kernel: UNIX-STREAM 4080KB 4080KB
Jun 23 18:12:29 client213 kernel: UNIX 4080KB 4080KB
Jun 23 18:12:29 client213 kernel: tcp_bind_bucket 1024KB 1024KB
Jun 23 18:12:29 client213 kernel: ip_fib_trie 195KB 195KB
Jun 23 18:12:29 client213 kernel: ip_fib_alias 195KB 195KB
Jun 23 18:12:29 client213 kernel: ip_dst_cache 1008KB 1008KB
Jun 23 18:12:29 client213 kernel: RAW 5280KB 5280KB
Jun 23 18:12:29 client213 kernel: UDP 4032KB 4032KB
Jun 23 18:12:29 client213 kernel: tw_sock_TCP 2046KB 2046KB
Jun 23 18:12:29 client213 kernel: request_sock_TCP 2014KB 2014KB
Jun 23 18:12:29 client213 kernel: TCP 4032KB 4032KB
Jun 23 18:12:29 client213 kernel: hugetlbfs_inode_cache 63KB 63KB
Jun 23 18:12:29 client213 kernel: bio-240 2064KB 2064KB
Jun 23 18:12:29 client213 kernel: ep_head 512KB 512KB
Jun 23 18:12:29 client213 kernel: eventpoll_pwq 512KB 512KB
Jun 23 18:12:29 client213 kernel: eventpoll_epi 1040KB 1040KB
Jun 23 18:12:29 client213 kernel: inotify_inode_mark 175KB 175KB
Jun 23 18:12:29 client213 kernel: request_queue 590KB 683KB
Jun 23 18:12:29 client213 kernel: blkdev_ioc 506KB 506KB
Jun 23 18:12:29 client213 kernel: bio-184 12938KB 12938KB
Jun 23 18:12:29 client213 kernel: biovec-max 10480KB 10624KB
Jun 23 18:12:29 client213 kernel: biovec-128 32KB 32KB
Jun 23 18:12:29 client213 kernel: biovec-64 4096KB 4096KB
Jun 23 18:12:29 client213 kernel: biovec-16 2048KB 2048KB
Jun 23 18:12:29 client213 kernel: bio_integrity_payload 7KB 7KB
Jun 23 18:12:29 client213 kernel: uid_cache 401KB 401KB
Jun 23 18:12:29 client213 kernel: dmaengine-unmap-256 30KB 30KB
Jun 23 18:12:29 client213 kernel: dmaengine-unmap-128 31KB 31KB
Jun 23 18:12:29 client213 kernel: dmaengine-unmap-16 7KB 7KB
Jun 23 18:12:29 client213 kernel: dmaengine-unmap-2 4KB 4KB
Jun 23 18:12:29 client213 kernel: audit_buffer 510KB 510KB
Jun 23 18:12:29 client213 kernel: skbuff_fclone_cache 8960KB 8960KB
Jun 23 18:12:29 client213 kernel: skbuff_head_cache 14560KB 14640KB
Jun 23 18:12:29 client213 kernel: configfs_dir_cache 3KB 3KB
Jun 23 18:12:29 client213 kernel: file_lock_cache 999KB 999KB
Jun 23 18:12:29 client213 kernel: file_lock_ctx 431KB 431KB
Jun 23 18:12:29 client213 kernel: fsnotify_mark_connector 148KB 148KB
Jun 23 18:12:29 client213 kernel: taskstats 2024KB 2024KB
Jun 23 18:12:29 client213 kernel: proc_dir_entry 3079KB 3079KB
Jun 23 18:12:29 client213 kernel: pde_opener 510KB 510KB
Jun 23 18:12:29 client213 kernel: seq_file 1020KB 1020KB
Jun 23 18:12:29 client213 kernel: sigqueue 510KB 510KB
Jun 23 18:12:29 client213 kernel: shmem_inode_cache 5850KB 5850KB
Jun 23 18:12:29 client213 kernel: kernfs_iattrs_cache 506KB 506KB
Jun 23 18:12:29 client213 kernel: kernfs_node_cache 25902KB 25920KB
Jun 23 18:12:29 client213 kernel: mnt_cache 2295KB 2295KB
Jun 23 18:12:29 client213 kernel: filp 15732KB 15824KB
Jun 23 18:12:29 client213 kernel: names_cache 9600KB 9664KB
Jun 23 18:12:29 client213 kernel: net_namespace 1251KB 1251KB
Jun 23 18:12:29 client213 kernel: hashtab_node 418KB 418KB
Jun 23 18:12:29 client213 kernel: ebitmap_node 3260KB 3260KB
Jun 23 18:12:29 client213 kernel: avtab_node 2139KB 2139KB
Jun 23 18:12:29 client213 kernel: avc_node 1145KB 1145KB
Jun 23 18:12:29 client213 kernel: lsm_inode_cache 6626KB 7360KB
Jun 23 18:12:29 client213 kernel: lsm_file_cache 516KB 516KB
Jun 23 18:12:29 client213 kernel: key_jar 2048KB 2048KB
Jun 23 18:12:29 client213 kernel: uts_namespace 62KB 62KB
Jun 23 18:12:29 client213 kernel: nsproxy 216KB 216KB
Jun 23 18:12:29 client213 kernel: vm_area_struct 13343KB 13343KB
Jun 23 18:12:29 client213 kernel: fs_cache 532KB 532KB
Jun 23 18:12:29 client213 kernel: files_cache 4269KB 4269KB
Jun 23 18:12:29 client213 kernel: signal_cache 13072KB 13104KB
Jun 23 18:12:29 client213 kernel: sighand_cache 14697KB 14757KB
Jun 23 18:12:29 client213 kernel: task_struct 38964KB 39128KB
Jun 23 18:12:29 client213 kernel: cred_jar 3417KB 3417KB
Jun 23 18:12:29 client213 kernel: anon_vma_chain 5992KB 6012KB
Jun 23 18:12:29 client213 kernel: anon_vma 5786KB 5790KB
Jun 23 18:12:29 client213 kernel: pid 11632KB 11720KB
Jun 23 18:12:29 client213 kernel: Acpi-Operand 34354KB 34354KB
Jun 23 18:12:29 client213 kernel: Acpi-ParseExt 75KB 75KB
Jun 23 18:12:29 client213 kernel: Acpi-Parse 75KB 75KB
Jun 23 18:12:29 client213 kernel: Acpi-State 75KB 75KB
Jun 23 18:12:29 client213 kernel: Acpi-Namespace 2633KB 2633KB
Jun 23 18:12:29 client213 kernel: numa_policy 47KB 47KB
Jun 23 18:12:29 client213 kernel: perf_event 4077KB 4077KB
Jun 23 18:12:29 client213 kernel: trace_event_file 253KB 253KB
Jun 23 18:12:29 client213 kernel: ftrace_event_field 294KB 294KB
Jun 23 18:12:29 client213 kernel: pool_workqueue 352KB 368KB
Jun 23 18:12:29 client213 kernel: task_group 4032KB 4032KB
Jun 23 18:12:29 client213 kernel: mm_struct 4335KB 4335KB
Jun 23 18:12:29 client213 kernel: vmap_area 1426KB 1568KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-8k 4096KB 4096KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-4k 7552KB 7936KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-2k 4704KB 4704KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-1k 5728KB 5728KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-512 26752KB 26912KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-256 1360KB 1360KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-192 1008KB 1008KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-128 1024KB 1024KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-96 504KB 504KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-64 512KB 512KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-32 568KB 568KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-16 512KB 512KB
Jun 23 18:12:30 client213 kernel: kmalloc-cg-8 512KB 512KB
Jun 23 18:12:30 client213 kernel: kmalloc-8k 130304KB 156960KB
Jun 23 18:12:30 client213 kernel: kmalloc-4k 26024KB 26444KB
Jun 23 18:12:30 client213 kernel: kmalloc-2k 22350KB 23944KB
Jun 23 18:12:30 client213 kernel: kmalloc-1k 48012KB 73184KB
Jun 23 18:12:30 client213 kernel: kmalloc-512 44522KB 260544KB
Jun 23 18:12:30 client213 kernel: kmalloc-256 18632KB 18720KB
Jun 23 18:12:30 client213 kernel: kmalloc-192 5745KB 7008KB
Jun 23 18:12:30 client213 kernel: kmalloc-128 6915KB 7660KB
Jun 23 18:12:30 client213 kernel: kmalloc-96 14678KB 14923KB
Jun 23 18:12:30 client213 kernel: kmalloc-64 378646970KB 378654316KB
Jun 23 18:12:30 client213 kernel: kmalloc-32 9460KB 10308KB
Jun 23 18:12:30 client213 kernel: kmalloc-16 4902KB 4924KB
Jun 23 18:12:30 client213 kernel: kmalloc-8 6365KB 6412KB
Jun 23 18:12:30 client213 kernel: kmem_cache_node 91KB 92KB
Jun 23 18:12:30 client213 kernel: kmem_cache 288KB 288KB
Jun 23 18:12:30 client213 kernel: Tasks state (memory values in pages):
Jun 23 18:12:30 client213 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jun 23 18:12:30 client213 kernel: [ 1781] 0 1781 40371 544 352256 0 -250 systemd-journal
Jun 23 18:12:30 client213 kernel: [ 1806] 0 1806 8666 989 86016 0 -1000 systemd-udevd
Jun 23 18:12:30 client213 kernel: [ 1994] 0 1994 1627 62 53248 0 0 rdma-ndd
Jun 23 18:12:30 client213 kernel: [ 2537] 0 2537 22996 241 69632 0 -1000 auditd
Jun 23 18:12:30 client213 kernel: [ 2539] 0 2539 1961 85 57344 0 0 sedispatch
Jun 23 18:12:30 client213 kernel: [ 2563] 0 2563 63738 808 131072 0 0 NetworkManager
Jun 23 18:12:30 client213 kernel: [ 2567] 0 2567 20241 530 61440 0 0 irqbalance
Jun 23 18:12:30 client213 kernel: [ 2568] 997 2568 690 45 40960 0 0 lsmd
Jun 23 18:12:30 client213 kernel: [ 2569] 0 2569 712 43 45056 0 0 mcelog
Jun 23 18:12:30 client213 kernel: [ 2570] 0 2570 168974 441 860160 0 0 rsyslogd
Jun 23 18:12:30 client213 kernel: [ 2571] 0 2571 2967 377 65536 0 0 smartd
Jun 23 18:12:31 client213 kernel: [ 2574] 991 2574 21109 163 61440 0 0 chronyd
Jun 23 18:12:31 client213 kernel: [ 2579] 0 2579 5113 655 90112 0 0 systemd-logind
Jun 23 18:12:31 client213 kernel: [ 2581] 81 2581 2714 173 61440 0 -900 dbus-broker-lau
Jun 23 18:12:31 client213 kernel: [ 2583] 81 2583 1282 131 49152 0 -900 dbus-broker
Jun 23 18:12:31 client213 kernel: [ 2588] 0 2588 4060 420 69632 0 -1000 sshd
Jun 23 18:12:31 client213 kernel: [ 2592] 0 2592 2949 84 45056 0 0 rshim
Jun 23 18:12:31 client213 kernel: [ 2697] 0 2697 1180 46 49152 0 0 atd
Jun 23 18:12:31 client213 kernel: [ 2704] 0 2704 1518 219 53248 0 0 crond
Jun 23 18:12:31 client213 kernel: [ 3224] 0 3224 761 21 45056 0 0 agetty
Jun 23 18:12:31 client213 kernel: [3065786] 0 3065786 5494 788 86016 0 100 systemd
Jun 23 18:12:31 client213 kernel: [3065788] 0 3065788 43393 1589 98304 0 100 (sd-pam)
Jun 23 18:12:31 client213 kernel: [3066324] 0 3066324 1050 113 45056 0 0 vdbench
Jun 23 18:12:31 client213 kernel: [3066352] 0 3066352 2548186 78649 2228224 0 0 java
Jun 23 18:12:31 client213 kernel: [3066514] 0 3066514 1047 100 40960 0 0 csh
Jun 23 18:12:31 client213 kernel: [3066537] 0 3066537 1050 113 53248 0 0 vdbench
Jun 23 18:12:31 client213 kernel: [3066559] 0 3066559 6332080 499888 6701056 0 0 java
Jun 23 18:12:31 client213 kernel: [3067965] 0 3067965 1050 114 45056 0 0 vdbench
Jun 23 18:12:31 client213 kernel: [3067993] 0 3067993 2548186 73860 2330624 0 0 java
Jun 23 18:12:31 client213 kernel: [3068161] 0 3068161 1047 102 40960 0 0 csh
Jun 23 18:12:31 client213 kernel: [3068184] 0 3068184 1050 113 45056 0 0 vdbench
Jun 23 18:12:31 client213 kernel: [3068206] 0 3068206 3555166 307004 4136960 0 0 java
Jun 23 18:12:31 client213 kernel: [3068833] 0 3068833 1278 183 49152 0 0 sh
Jun 23 18:12:31 client213 kernel: [1110867] 0 1110867 1050 113 45056 0 0 vdbench
Jun 23 18:12:31 client213 kernel: [1110895] 0 1110895 2548186 62778 2191360 0 0 java
Jun 23 18:12:31 client213 kernel: [1111063] 0 1111063 1047 102 45056 0 0 csh
Jun 23 18:12:31 client213 kernel: [1111086] 0 1111086 1050 114 45056 0 0 vdbench
Jun 23 18:12:31 client213 kernel: [1111108] 0 1111108 9349668 446043 7544832 0 0 java
Jun 23 18:12:31 client213 kernel: [1189194] 0 1189194 3396 434 69632 0 0 crond
Jun 23 18:12:31 client213 kernel: [1189199] 0 1189199 1150 61 45056 0 0 sh
Jun 23 18:12:31 client213 kernel: [1189200] 0 1189200 1150 73 49152 0 0 sh
Jun 23 18:12:31 client213 kernel: [1189210] 0 1189210 1050 114 45056 0 0 vdbench
Jun 23 18:12:31 client213 kernel: [1189237] 0 1189237 2548186 40600 1810432 0 0 java
Jun 23 18:12:31 client213 kernel: [1189239] 0 1189239 765 22 45056 0 0 sleep
Jun 23 18:12:31 client213 kernel: [1189400] 0 1189400 1047 102 49152 0 0 csh
Jun 23 18:12:31 client213 kernel: [1189423] 0 1189423 1050 113 45056 0 0 vdbench
Jun 23 18:12:31 client213 kernel: [1189445] 0 1189445 16046686 522122 11182080 0 0 java
Jun 23 18:12:31 client213 kernel: [1190719] 0 1190719 765 22 40960 0 0 sleep
Jun 23 18:12:31 client213 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/user.slice/user-0.slice/user@0.service,task=(sd-pam),pid=3065788,uid=0
Jun 23 18:12:31 client213 kernel: Out of memory: Killed process 3065788 ((sd-pam)) total-vm:173572kB, anon-rss:6356kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:96kB oom_score_adj:100
Jun 23 18:13:01 client213 kernel: obd_memory max: 35736380442, obd_memory current: 109177019
{{}}