Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
Lustre 2.4.3
-
None
-
3
-
16909
Description
We have had a number of crashes with the MDS OOMing with ldlm_locks slab using most of the memory. Attached you'll find console logs and back trace.
<code>
rash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 12289376 46.9 GB ----
FREE 348961 1.3 GB 2% of TOTAL MEM
USED 11940415 45.5 GB 97% of TOTAL MEM
SHARED 251654 983 MB 2% of TOTAL MEM
BUFFERS 250789 979.6 MB 2% of TOTAL MEM
CACHED 864 3.4 MB 0% of TOTAL MEM
SLAB 9196563 35.1 GB 74% of TOTAL MEM
TOTAL SWAP 500013 1.9 GB ----
SWAP USED 2913 11.4 MB 0% of TOTAL SWAP
SWAP FREE 497100 1.9 GB 99% of TOTAL SWAP
crash> kmem -s
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
ffff880ba5641980 osp_obj 216 38190 85302 4739 4k
ffff880babc51940 lod_obj 120 21141 64096 2003 4k
ffff880bb1951900 mdt_obj 248 21141 54528 3408 4k
ffff880bb34a18c0 fsfilt_ldiskfs_fcb 56 0 0 0 4k
ffff880bb3631880 dynlock_cache 128 0 0 0 4k
ffff880bb3621840 ldiskfs_inode_cache 1056 22187 33852 11284 4k
ffff880bb3611800 ldiskfs_xattr 88 0 0 0 4k
ffff880bb36017c0 ldiskfs_free_data 64 0 0 0 4k
ffff880bb35f1780 ldiskfs_alloc_context 136 0 0 0 4k
ffff880bb35e1740 ldiskfs_prealloc_space 112 37 170 5 4k
ffff880bb35d1700 ldiskfs_system_zone 40 0 0 0 4k
ffff880bb35516c0 upd_kmem 96 0 0 0 4k
ffff880bb3541680 lqe_kmem 192 3130 3180 159 4k
ffff880bb3491640 jbd2_journal_handle 48 0 0 0 4k
ffff880bb3481600 jbd2_journal_head 112 0 0 0 4k
ffff880bb3b715c0 jbd2_revoke_table 16 4 404 2 4k
ffff880bb3b81580 jbd2_revoke_record 32 0 0 0 4k
ffff880bb3461540 mdd_obj 96 21141 68200 1705 4k
ffff8805fd5a2040 ccc_req_kmem 40 0 0 0 4k
ffff8805fd592000 ccc_session_kmem 184 589 1890 90 4k
ffff8805fd581fc0 ccc_thread_kmem 352 71 176 16 4k
ffff8805fdfb1f80 ccc_object_kmem 264 0 0 0 4k
ffff8805fdfa1f40 ccc_lock_kmem 40 0 0 0 4k
ffff8805fdf91f00 vvp_session_kmem 104 589 2183 59 4k
ffff8805fdf81ec0 vvp_thread_kmem 488 71 136 17 4k
ffff8805fde31e80 ll_rmtperm_hash_cache 256 0 0 0 4k
ffff8805fde21e40 ll_remote_perm_cache 40 0 0 0 4k
ffff8805fe391e00 ll_file_data 192 0 0 0 4k
ffff880601741dc0 lustre_inode_cache 1216 0 0 0 4k
ffff8805fdf71d80 lov_oinfo 128 0 0 0 4k
ffff8805fdf61d40 lov_lock_link_kmem 32 0 0 0 4k
ffff8805fdf51d00 lovsub_req_kmem 40 0 0 0 4k
ffff8805fdf41cc0 lovsub_object_kmem 240 0 0 0 4k
ffff8805fdf31c80 lovsub_lock_kmem 64 0 0 0 4k
ffff8805fdf21c40 lov_req_kmem 40 0 0 0 4k
ffff8805fdd11c00 lov_session_kmem 400 589 1110 111 4k
ffff8805fdd01bc0 lov_thread_kmem 288 71 195 15 4k
ffff8805fdf11b80 lov_object_kmem 240 0 0 0 4k
ffff8805fdcf1b40 lov_lock_kmem 104 0 0 0 4k
ffff8805fde11b00 osc_quota_kmem 24 0 0 0 4k
ffff8805fde01ac0 osc_extent_kmem 168 0 0 0 4k
ffff8805fddf1a80 osc_req_kmem 40 0 0 0 4k
ffff8805fdde1a40 osc_session_kmem 424 589 1080 120 4k
ffff8805fddd1a00 osc_thread_kmem 984 71 96 24 4k
ffff8805fddc19c0 osc_object_kmem 288 0 0 0 4k
ffff8805fddb1980 osc_lock_kmem 192 0 0 0 4k
ffff8805fe371940 interval_node 128 0 0 0 4k
ffff8805fe361900 ldlm_locks 576 49731039 49796635 7113805 4k
<cod>