Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
None
-
3
-
9223372036854775807
Description
This issue was created by maloo for Andreas Dilger <adilger@whamcloud.com>
This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/6815c7d5-e235-4503-a7f9-2ec3a7066e77
test_56b failed with the following error:
trevis-77vm3 crashed during sanityn test_56b
Have started to hit OOM in the new sanity test_56b (llverdev) with ZFS since this test was landed via patch https://review.whamcloud.com/43230 "LU-8137 utils: fix llverdev for use on regular files". The test is writing a $((16TB * OSTCOUNT)) = 128TB sparse 2000-overstriped file with 4KB verification chunks every 1MB offset. That should be a total of 64MB/OST, and $((2000 / OSTCOUNT)) = 128 objects per OST.
Despite the name, llverdev is just a userspace application writing to a regular file in this case. The test is intended to verify the correct handling of very large files (16TB, though not 16TiB OST objects, and 100TB+ total file size).
The osd-zfs code should be handling the sparse 4KB writes by setting the ZFS blocksize = 4KB for this file (per patch http://review.whamcloud.com/15127 "LU-4865 zfs: grow block size by write pattern"), but I haven't looked into that to confirm. It might be inflating the 4KB writes to 128KB or even 1MB blocksize, but that shouldn't be happening and would be a bug in osd-zfs separate from the OOM.
The OSS console shows:
[16393.499280] Lustre: ll_ost00_029: service thread pid 263248 was inactive for 43.423 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [16393.506264] Pid: 263248, comm: ll_ost00_029 4.18.0-348.23.1.el8_lustre.x86_64 #1 SMP Mon May 30 23:09:30 UTC 2022 [16393.508375] Call Trace TBD: [16393.509368] [<0>] __cv_timedwait_common+0x12e/0x160 [spl] [16393.510609] [<0>] __cv_timedwait_io+0x15/0x20 [spl] [16393.511744] [<0>] zio_wait+0x133/0x270 [zfs] [16393.512783] [<0>] dmu_tx_hold_free_impl+0x1fc/0x250 [zfs] [16393.514104] [<0>] dmu_free_long_range+0x22e/0x4c0 [zfs] [16393.515439] [<0>] osd_unlinked_object_free+0x44/0x390 [osd_zfs] [16393.516785] [<0>] osd_unlinked_list_emptify+0xa6/0xb0 [osd_zfs] [16393.518173] [<0>] osd_trans_stop+0x35c/0x570 [osd_zfs] [16393.519532] [<0>] ofd_destroy+0x3b4/0xb20 [ofd] [16393.520416] [<0>] ofd_destroy_by_fid+0x25a/0x4a0 [ofd] [16393.521531] [<0>] ofd_destroy_hdl+0x263/0xa10 [ofd] [16393.523296] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc] [16393.524598] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc] [16393.526294] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc] [16393.527318] [<0>] kthread+0x116/0x130 [16393.528121] [<0>] ret_from_fork+0x35/0x40 [16506.409968] obd_memory max: 200336259, obd_memory current: 200336259 [16506.975974] obd_memory max: 200416739, obd_memory current: 200416739 [16507.013294] obd_memory max: 200416739, obd_memory current: 200416739 [16507.020553] obd_memory max: 200416739, obd_memory current: 200416739 [16507.035227] obd_memory max: 200416739, obd_memory current: 200416739 [16507.218562] obd_memory max: 200471595, obd_memory current: 200471595 [16507.224060] obd_memory max: 200471595, obd_memory current: 200471595 [16507.226494] obd_memory max: 200471595, obd_memory current: 200471595 [16507.229583] obd_memory max: 200471595, obd_memory current: 200471595 [16507.231476] obd_memory max: 200471595, obd_memory current: 200471595 [16507.232739] XXXX invoked oom-killer: gfp_mask=0x6002c0(GFP_KERNEL|__GFP_NOWARN), order=0, oom_score_adj=0 <different processes actually trigger the OOM>
and the memory dump looks like:
[16162.520995] Mem-Info: [16162.521505] active_anon:7 inactive_anon:6 isolated_anon:0 active_file:27 inactive_file:15 isolated_file:0 unevictable:0 dirty:0 writeback:0 slab_reclaimable:5000 slab_unreclaimable:101763 mapped:259 shmem:6 pagetables:1721 bounce:0 free:11251 free_pcp:0 free_cma:0 [16162.526963] Node 0 active_anon:28kB inactive_anon:24kB active_file:108kB inactive_file:60kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1036kB dirty:0kB writeback:0kB shmem:24kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:9440kB pagetables:6884kB all_unreclaimable? no [16162.532034] Node 0 DMA free:10688kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [16162.536302] lowmem_reserve[]: 0 2617 2617 2617 2617 [16162.537266] Node 0 DMA32 free:34820kB min:44796kB low:55992kB high:67188kB active_anon:28kB inactive_anon:24kB active_file:0kB inactive_file:168kB unevictable:0kB writepending:0kB present:3129320kB managed:2713588kB mlocked:0kB bounce:0kB free_pcp:332kB local_pcp:0kB free_cma:0kB [16162.541692] lowmem_reserve[]: 0 0 0 0 0 [16162.542461] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 1*32kB (U) 0*64kB 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (M) 2*4096kB (M) = 10688kB [16162.544804] Node 0 DMA32: 845*4kB (UME) 1044*8kB (UMEH) 191*16kB (UMEH) 139*32kB (UMEH) 36*64kB (UMEH) 13*128kB (UMH) 6*256kB (UM) 7*512kB (MH) 3*1024kB (MH) 2*2048kB (M) 0*4096kB = 35492kB [16162.547881] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [16162.549501] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [16162.551073] 46 total pagecache pages [16162.551793] 1 pages in swap cache [16162.552481] Swap cache stats: add 54069, delete 54066, find 346/1097 [16162.553695] Free swap = 2648636kB [16162.554380] Total swap = 2860028kB [16162.555077] 786328 pages RAM [16162.555708] 0 pages HighMem/MovableOnly [16162.556562] 104091 pages reserved [16162.557249] 0 pages hwpoisoned [16162.557882] Unreclaimable slab info: [16162.558596] Name Used Total [16162.559660] ofd_obj 82KB 144KB [16162.560668] osd_zapit_cache 15KB 15KB [16162.561674] zfs_osd_obj 152KB 324KB [16162.562677] vvp_thread_kmem 15KB 15KB [16162.563676] vvp_session_kmem 7KB 7KB [16162.564678] ll_thread_kmem 7KB 7KB [16162.565716] lov_session_kmem 15KB 15KB [16162.566726] lov_thread_kmem 7KB 7KB [16162.567734] osc_session_kmem 7KB 7KB [16162.568758] osc_thread_kmem 60KB 60KB [16162.569766] ldlm_glimpse_work_kmem 7KB 7KB [16162.570844] interval_tree 256KB 256KB [16162.571844] interval_node 264KB 264KB [16162.572853] ldlm_locks 1167KB 1181KB [16162.573861] ldlm_resources 385KB 385KB [16162.574863] lnet_msg 108KB 108KB [16162.575865] lnet_rspt 135KB 135KB [16162.576871] lnet_small_MDs 51KB 51KB [16162.577876] lnet_MEs 71KB 71KB [16162.578889] ptlrpc_cache 446KB 677KB [16162.579890] tgt_fmd_cache 63KB 63KB [16162.580896] tgt_session_kmem 114KB 114KB [16162.581900] tgt_thread_kmem 277KB 277KB [16162.582921] cl_env_kmem 3KB 3KB [16162.583933] ll_obd_dev_cache 283KB 283KB [16162.584948] zfs_znode_hold_cache 7KB 7KB [16162.585999] zfs_znode_cache 31KB 31KB [16162.587036] zil_zcw_cache 7KB 7KB [16162.588043] zil_lwb_cache 7KB 7KB [16162.589172] dmu_buf_impl_t 9854KB 12139KB [16162.590213] arc_buf_t 1856KB 2338KB [16162.591373] arc_buf_hdr_t_full 7418KB 9337KB [16162.592419] dnode_t 4028KB 5378KB [16162.593438] sa_cache 86KB 168KB [16162.594479] abd_t 1893KB 2570KB [16162.595497] lz4_cache 64KB 64KB [16162.596514] zio_buf_comb_16384 4512KB 5504KB [16162.597533] zio_buf_comb_12288 48KB 48KB [16162.598576] zio_buf_comb_10240 48KB 48KB [16162.599595] zio_buf_comb_8192 160KB 352KB [16162.600613] zio_buf_comb_7168 64KB 64KB [16162.601620] zio_buf_comb_6144 64KB 64KB [16162.602620] zio_buf_comb_5120 64KB 64KB [16162.603742] zio_buf_comb_4096 37076KB 53472KB [16162.604759] zio_buf_comb_3584 63KB 63KB [16162.605792] zio_buf_comb_3072 60KB 60KB [16162.606811] zio_buf_comb_2560 60KB 60KB [16162.607840] zio_buf_comb_2048 64KB 64KB [16162.608852] zio_buf_comb_1536 126KB 126KB [16162.609858] zio_buf_comb_1024 480KB 832KB [16162.610870] zio_buf_comb_512 193KB 304KB [16162.611882] zio_link_cache 804KB 1051KB [16162.612970] zio_cache 19182KB 26475KB [16162.614139] zfs_btree_leaf_cache 2860KB 3200KB [16162.615275] mod_hash_entries 3KB 3KB [16162.616291] nfs_commit_data 15KB 15KB [16162.617309] nfs_write_data 31KB 31KB [16162.618321] nfs_read_data 15KB 15KB [16162.619335] nfs_page 4KB 4KB [16162.620346] fscache_cookie_jar 7KB 7KB [16162.621417] rpc_buffers 64KB 64KB [16162.622435] rpc_tasks 8KB 8KB [16162.623449] bio-208 48KB 48KB [16162.624488] bio-272 97KB 97KB [16162.625526] ext4_system_zone 3KB 3KB [16162.626714] jbd2_inode 40KB 40KB [16162.627732] scsi_sense_cache 4KB 4KB [16162.628739] bio-160 11KB 11KB [16162.629755] fib6_nodes 4KB 4KB [16162.630772] ip6_dst_cache 7KB 7KB [16162.631792] RAWv6 15KB 15KB [16162.632810] UDPv6 94KB 94KB [16162.633814] TCPv6 61KB 61KB [16162.634833] sgpool-128 32KB 32KB [16162.635839] sgpool-64 32KB 32KB [16162.636875] sgpool-32 16KB 16KB [16162.637906] sgpool-16 8KB 8KB [16162.638915] sgpool-8 4KB 4KB [16162.639923] mqueue_inode_cache 16KB 16KB [16162.640952] kioctx 31KB 31KB [16162.641960] aio_kiocb 7KB 7KB [16162.642972] bio-248 8KB 8KB [16162.643986] posix_timers_cache 19KB 19KB [16162.645008] UNIX 321KB 330KB [16162.646015] tcp_bind_bucket 16KB 16KB [16162.647053] ip_fib_trie 7KB 7KB [16162.648061] ip_fib_alias 7KB 7KB [16162.649077] ip_dst_cache 56KB 72KB [16162.650093] RAW 15KB 15KB [16162.651103] UDP 45KB 45KB [16162.652115] tw_sock_TCP 19KB 19KB [16162.653128] request_sock_TCP 7KB 7KB [16162.654142] TCP 95KB 95KB [16162.655161] hugetlbfs_inode_cache 7KB 7KB [16162.656238] bio-288 11KB 11KB [16162.657248] eventpoll_pwq 23KB 23KB [16162.658260] eventpoll_epi 68KB 68KB [16162.659274] inotify_inode_mark 7KB 7KB [16162.660298] request_queue 63KB 63KB [16162.661311] blkdev_ioc 35KB 35KB [16162.662324] bio-224 76KB 76KB [16162.663333] biovec-max 544KB 704KB [16162.664343] biovec-128 64KB 64KB [16162.665355] biovec-64 48KB 48KB [16162.666362] biovec-16 8KB 8KB [16162.667375] bio_integrity_payload 4KB 4KB [16162.668458] khugepaged_mm_slot 7KB 7KB [16162.669486] uid_cache 7KB 7KB [16162.670501] dmaengine-unmap-256 30KB 30KB [16162.671569] dmaengine-unmap-128 15KB 15KB [16162.672624] dmaengine-unmap-16 3KB 3KB [16162.673673] dmaengine-unmap-2 4KB 4KB [16162.674678] audit_buffer 7KB 7KB [16162.675684] skbuff_fclone_cache 70KB 70KB [16162.676723] skbuff_head_cache 28KB 32KB [16162.677721] configfs_dir_cache 3KB 3KB [16162.678735] file_lock_cache 7KB 7KB [16162.679735] file_lock_ctx 7KB 7KB [16162.680745] fsnotify_mark_connector 7KB 7KB [16162.681840] net_namespace 31KB 31KB [16162.682844] task_delay_info 87KB 87KB [16162.683858] taskstats 15KB 15KB [16162.684863] proc_dir_entry 252KB 252KB [16162.685861] pde_opener 7KB 7KB [16162.686894] seq_file 8KB 8KB [16162.687903] sigqueue 7KB 7KB [16162.688935] shmem_inode_cache 845KB 882KB [16162.689942] kernfs_iattrs_cache 15KB 15KB [16162.690984] kernfs_node_cache 4063KB 4107KB [16162.691989] mnt_cache 252KB 252KB [16162.693004] filp 474KB 536KB [16162.694011] names_cache 64KB 64KB [16162.695015] avc_node 7KB 7KB [16162.696040] lsm_inode_cache 1102KB 1692KB [16162.697057] lsm_file_cache 120KB 120KB [16162.698063] key_jar 28KB 28KB [16162.699072] nsproxy 7KB 7KB [16162.700083] vm_area_struct 1271KB 1375KB [16162.701092] mm_struct 135KB 157KB [16162.702101] fs_cache 8KB 8KB [16162.703110] files_cache 94KB 94KB [16162.704275] signal_cache 699KB 725KB [16162.705290] sighand_cache 1206KB 1206KB [16162.706300] task_struct 3580KB 3651KB [16162.707316] cred_jar 158KB 169KB [16162.708337] anon_vma_chain 223KB 268KB [16162.709348] anon_vma 240KB 249KB [16162.710360] pid 184KB 184KB [16162.711374] Acpi-Operand 74KB 74KB [16162.712402] Acpi-ParseExt 3KB 3KB [16162.713415] Acpi-Parse 3KB 3KB [16162.714422] Acpi-State 11KB 11KB [16162.715455] Acpi-Namespace 15KB 15KB [16162.716466] numa_policy 3KB 3KB [16162.717475] trace_event_file 150KB 150KB [16162.718483] ftrace_event_field 203KB 203KB [16162.719512] pool_workqueue 12KB 12KB [16162.720519] task_group 63KB 63KB [16162.721526] vmap_area 388KB 392KB [16162.722550] kmalloc-cg-4k 96KB 96KB [16162.723562] kmalloc-cg-2k 160KB 160KB [16162.724567] kmalloc-cg-1k 152KB 192KB [16162.725582] kmalloc-cg-512 24KB 24KB [16162.726585] kmalloc-cg-192 23KB 23KB [16162.727604] kmalloc-cg-96 7KB 7KB [16162.728609] kmalloc-cg-64 8KB 8KB [16162.729619] kmalloc-cg-32 8KB 8KB [16162.730633] kmalloc-cg-16 8KB 8KB [16162.731635] kmalloc-cg-8 8KB 8KB [16162.732648] kmalloc-8k 4760KB 4896KB [16162.733652] kmalloc-4k 10980KB 11008KB [16162.734662] kmalloc-2k 3424KB 3488KB [16162.735681] kmalloc-1k 28142KB 28348KB [16162.736808] kmalloc-512 14760KB 16316KB [16162.737832] kmalloc-256 577KB 624KB [16162.738877] kmalloc-192 2397KB 2795KB [16162.739938] kmalloc-128 635KB 920KB [16162.740952] kmalloc-96 2893KB 2909KB [16162.741993] kmalloc-64 2310KB 2668KB [16162.743025] kmalloc-32 1790KB 1816KB [16162.744083] kmalloc-16 164KB 164KB [16162.745099] kmalloc-8 52KB 52KB [16162.746110] kmem_cache_node 24KB 24KB [16162.747117] kmem_cache 149KB 149KB [16162.748125] Tasks state (memory values in pages): [16162.749030] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [16162.750628] [ 560] 0 560 30725 0 311296 530 0 systemd-journal [16162.752299] [ 598] 0 598 28690 0 217088 325 -1000 systemd-udevd [16162.753946] [ 733] 32 733 16800 0 172032 195 0 rpcbind [16162.755492] [ 737] 0 737 19253 0 155648 169 -1000 auditd [16162.757030] [ 765] 70 765 21366 0 176128 205 0 avahi-daemon [16162.758652] [ 766] 81 766 15815 0 176128 231 -900 dbus-daemon [16162.760255] [ 767] 0 767 31223 0 147456 185 0 irqbalance [16162.761851] [ 769] 0 769 54776 0 442368 536 0 sssd [16162.763357] [ 770] 0 770 97977 0 380928 1203 0 NetworkManager [16162.765014] [ 771] 998 771 407557 0 331776 1788 0 polkitd [16162.766552] [ 773] 995 773 37789 0 188416 196 0 chronyd [16162.768111] [ 775] 70 775 21302 0 151552 109 0 avahi-daemon [16162.769733] [ 790] 0 790 23087 1 192512 233 -1000 sshd [16162.771239] [ 791] 0 791 124104 58 446464 4701 0 tuned [16162.772792] [ 802] 994 802 31995 0 147456 184 0 munged [16162.774338] [ 804] 0 804 15005 0 151552 148 0 xinetd [16162.775905] [ 807] 0 807 57024 0 450560 809 0 sssd_be [16162.777453] [ 825] 0 825 97428 0 278528 510 0 rsyslogd [16162.779027] [ 838] 29 838 17942 0 176128 6297 0 rpc.statd [16162.780607] [ 859] 0 859 57317 0 471040 481 0 sssd_nss [16162.782172] [ 880] 0 880 25444 0 176128 175 0 gssproxy [16162.783743] [ 893] 0 893 34690 0 286720 366 0 cupsd [16162.785290] [ 897] 0 897 23157 0 208896 257 0 systemd-logind [16162.786967] [ 1045] 0 1045 30781 0 192512 255 0 master [16162.788493] [ 1057] 89 1057 37618 0 237568 274 0 qmgr [16162.790001] [ 1100] 0 1100 102309 0 294912 472 0 automount [16162.791569] [ 1106] 0 1106 9239 0 106496 242 0 crond [16162.793101] [ 1108] 0 1108 11001 0 110592 54 0 atd [16162.794586] [ 1126] 0 1126 3414 0 65536 33 0 agetty [16162.796153] [ 1127] 0 1127 4056 0 69632 37 0 agetty [16162.797869] [ 279880] 89 279880 37605 0 233472 276 0 pickup [16162.799424] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanityn test_56b - trevis-77vm3 crashed during sanityn test_56b