[LU-15963] sanityn test_56b: OSS OOM with ZFS

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Fix Version/s: Lustre 2.17.0
• Affects Version/s: Lustre 2.16.0
• Labels: zfs
• Severity: 3
• Assignee: Core Lustre Triage
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/6815c7d5-e235-4503-a7f9-2ec3a7066e77

test_56b failed with the following error:

trevis-77vm3 crashed during sanityn test_56b

Have started to hit OOM in the new sanity test_56b (llverdev) with ZFS since this test was landed via patch https://review.whamcloud.com/43230 "LU-8137 utils: fix llverdev for use on regular files". The test is writing a $((16TB * OSTCOUNT)) = 128TB sparse 2000-overstriped file with 4KB verification chunks every 1MB offset. That should be a total of 64MB/OST, and $((2000 / OSTCOUNT)) = 128 objects per OST.

Despite the name, llverdev is just a userspace application writing to a regular file in this case. The test is intended to verify the correct handling of very large files (16TB, though not 16TiB OST objects, and 100TB+ total file size).

The osd-zfs code should be handling the sparse 4KB writes by setting the ZFS blocksize = 4KB for this file (per patch http://review.whamcloud.com/15127 "LU-4865 zfs: grow block size by write pattern"), but I haven't looked into that to confirm. It might be inflating the 4KB writes to 128KB or even 1MB blocksize, but that shouldn't be happening and would be a bug in osd-zfs separate from the OOM.

The OSS console shows:

[16393.499280] Lustre: ll_ost00_029: service thread pid 263248 was inactive for 43.423 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[16393.506264] Pid: 263248, comm: ll_ost00_029 4.18.0-348.23.1.el8_lustre.x86_64 #1 SMP Mon May 30 23:09:30 UTC 2022
[16393.508375] Call Trace TBD:
[16393.509368] [<0>] __cv_timedwait_common+0x12e/0x160 [spl]
[16393.510609] [<0>] __cv_timedwait_io+0x15/0x20 [spl]
[16393.511744] [<0>] zio_wait+0x133/0x270 [zfs]
[16393.512783] [<0>] dmu_tx_hold_free_impl+0x1fc/0x250 [zfs]
[16393.514104] [<0>] dmu_free_long_range+0x22e/0x4c0 [zfs]
[16393.515439] [<0>] osd_unlinked_object_free+0x44/0x390 [osd_zfs]
[16393.516785] [<0>] osd_unlinked_list_emptify+0xa6/0xb0 [osd_zfs]
[16393.518173] [<0>] osd_trans_stop+0x35c/0x570 [osd_zfs]
[16393.519532] [<0>] ofd_destroy+0x3b4/0xb20 [ofd]
[16393.520416] [<0>] ofd_destroy_by_fid+0x25a/0x4a0 [ofd]
[16393.521531] [<0>] ofd_destroy_hdl+0x263/0xa10 [ofd]
[16393.523296] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
[16393.524598] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
[16393.526294] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
[16393.527318] [<0>] kthread+0x116/0x130
[16393.528121] [<0>] ret_from_fork+0x35/0x40
[16506.409968] obd_memory max: 200336259, obd_memory current: 200336259
[16506.975974] obd_memory max: 200416739, obd_memory current: 200416739
[16507.013294] obd_memory max: 200416739, obd_memory current: 200416739
[16507.020553] obd_memory max: 200416739, obd_memory current: 200416739
[16507.035227] obd_memory max: 200416739, obd_memory current: 200416739
[16507.218562] obd_memory max: 200471595, obd_memory current: 200471595
[16507.224060] obd_memory max: 200471595, obd_memory current: 200471595
[16507.226494] obd_memory max: 200471595, obd_memory current: 200471595
[16507.229583] obd_memory max: 200471595, obd_memory current: 200471595
[16507.231476] obd_memory max: 200471595, obd_memory current: 200471595
[16507.232739] XXXX invoked oom-killer: gfp_mask=0x6002c0(GFP_KERNEL|__GFP_NOWARN), order=0, oom_score_adj=0
<different processes actually trigger the OOM>

and the memory dump looks like:

[16162.520995] Mem-Info:
[16162.521505] active_anon:7 inactive_anon:6 isolated_anon:0
 active_file:27 inactive_file:15 isolated_file:0
 unevictable:0 dirty:0 writeback:0
 slab_reclaimable:5000 slab_unreclaimable:101763
 mapped:259 shmem:6 pagetables:1721 bounce:0
 free:11251 free_pcp:0 free_cma:0
[16162.526963] Node 0 active_anon:28kB inactive_anon:24kB active_file:108kB inactive_file:60kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1036kB dirty:0kB writeback:0kB shmem:24kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:9440kB pagetables:6884kB all_unreclaimable? no
[16162.532034] Node 0 DMA free:10688kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[16162.536302] lowmem_reserve[]: 0 2617 2617 2617 2617
[16162.537266] Node 0 DMA32 free:34820kB min:44796kB low:55992kB high:67188kB active_anon:28kB inactive_anon:24kB active_file:0kB inactive_file:168kB unevictable:0kB writepending:0kB present:3129320kB managed:2713588kB mlocked:0kB bounce:0kB free_pcp:332kB local_pcp:0kB free_cma:0kB
[16162.541692] lowmem_reserve[]: 0 0 0 0 0
[16162.542461] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 1*32kB (U) 0*64kB 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (M) 2*4096kB (M) = 10688kB
[16162.544804] Node 0 DMA32: 845*4kB (UME) 1044*8kB (UMEH) 191*16kB (UMEH) 139*32kB (UMEH) 36*64kB (UMEH) 13*128kB (UMH) 6*256kB (UM) 7*512kB (MH) 3*1024kB (MH) 2*2048kB (M) 0*4096kB = 35492kB
[16162.547881] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[16162.549501] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[16162.551073] 46 total pagecache pages
[16162.551793] 1 pages in swap cache
[16162.552481] Swap cache stats: add 54069, delete 54066, find 346/1097
[16162.553695] Free swap  = 2648636kB
[16162.554380] Total swap = 2860028kB
[16162.555077] 786328 pages RAM
[16162.555708] 0 pages HighMem/MovableOnly
[16162.556562] 104091 pages reserved
[16162.557249] 0 pages hwpoisoned
[16162.557882] Unreclaimable slab info:
[16162.558596] Name                      Used          Total
[16162.559660] ofd_obj                   82KB        144KB
[16162.560668] osd_zapit_cache           15KB         15KB
[16162.561674] zfs_osd_obj              152KB        324KB
[16162.562677] vvp_thread_kmem           15KB         15KB
[16162.563676] vvp_session_kmem           7KB          7KB
[16162.564678] ll_thread_kmem             7KB          7KB
[16162.565716] lov_session_kmem          15KB         15KB
[16162.566726] lov_thread_kmem            7KB          7KB
[16162.567734] osc_session_kmem           7KB          7KB
[16162.568758] osc_thread_kmem           60KB         60KB
[16162.569766] ldlm_glimpse_work_kmem          7KB          7KB
[16162.570844] interval_tree            256KB        256KB
[16162.571844] interval_node            264KB        264KB
[16162.572853] ldlm_locks              1167KB       1181KB
[16162.573861] ldlm_resources           385KB        385KB
[16162.574863] lnet_msg                 108KB        108KB
[16162.575865] lnet_rspt                135KB        135KB
[16162.576871] lnet_small_MDs            51KB         51KB
[16162.577876] lnet_MEs                  71KB         71KB
[16162.578889] ptlrpc_cache             446KB        677KB
[16162.579890] tgt_fmd_cache             63KB         63KB
[16162.580896] tgt_session_kmem         114KB        114KB
[16162.581900] tgt_thread_kmem          277KB        277KB
[16162.582921] cl_env_kmem                3KB          3KB
[16162.583933] ll_obd_dev_cache         283KB        283KB
[16162.584948] zfs_znode_hold_cache          7KB          7KB
[16162.585999] zfs_znode_cache           31KB         31KB
[16162.587036] zil_zcw_cache              7KB          7KB
[16162.588043] zil_lwb_cache              7KB          7KB
[16162.589172] dmu_buf_impl_t          9854KB      12139KB
[16162.590213] arc_buf_t               1856KB       2338KB
[16162.591373] arc_buf_hdr_t_full       7418KB       9337KB
[16162.592419] dnode_t                 4028KB       5378KB
[16162.593438] sa_cache                  86KB        168KB
[16162.594479] abd_t                   1893KB       2570KB
[16162.595497] lz4_cache                 64KB         64KB
[16162.596514] zio_buf_comb_16384       4512KB       5504KB
[16162.597533] zio_buf_comb_12288         48KB         48KB
[16162.598576] zio_buf_comb_10240         48KB         48KB
[16162.599595] zio_buf_comb_8192        160KB        352KB
[16162.600613] zio_buf_comb_7168         64KB         64KB
[16162.601620] zio_buf_comb_6144         64KB         64KB
[16162.602620] zio_buf_comb_5120         64KB         64KB
[16162.603742] zio_buf_comb_4096      37076KB      53472KB
[16162.604759] zio_buf_comb_3584         63KB         63KB
[16162.605792] zio_buf_comb_3072         60KB         60KB
[16162.606811] zio_buf_comb_2560         60KB         60KB
[16162.607840] zio_buf_comb_2048         64KB         64KB
[16162.608852] zio_buf_comb_1536        126KB        126KB
[16162.609858] zio_buf_comb_1024        480KB        832KB
[16162.610870] zio_buf_comb_512         193KB        304KB
[16162.611882] zio_link_cache           804KB       1051KB
[16162.612970] zio_cache              19182KB      26475KB
[16162.614139] zfs_btree_leaf_cache       2860KB       3200KB
[16162.615275] mod_hash_entries           3KB          3KB
[16162.616291] nfs_commit_data           15KB         15KB
[16162.617309] nfs_write_data            31KB         31KB
[16162.618321] nfs_read_data             15KB         15KB
[16162.619335] nfs_page                   4KB          4KB
[16162.620346] fscache_cookie_jar          7KB          7KB
[16162.621417] rpc_buffers               64KB         64KB
[16162.622435] rpc_tasks                  8KB          8KB
[16162.623449] bio-208                   48KB         48KB
[16162.624488] bio-272                   97KB         97KB
[16162.625526] ext4_system_zone           3KB          3KB
[16162.626714] jbd2_inode                40KB         40KB
[16162.627732] scsi_sense_cache           4KB          4KB
[16162.628739] bio-160                   11KB         11KB
[16162.629755] fib6_nodes                 4KB          4KB
[16162.630772] ip6_dst_cache              7KB          7KB
[16162.631792] RAWv6                     15KB         15KB
[16162.632810] UDPv6                     94KB         94KB
[16162.633814] TCPv6                     61KB         61KB
[16162.634833] sgpool-128                32KB         32KB
[16162.635839] sgpool-64                 32KB         32KB
[16162.636875] sgpool-32                 16KB         16KB
[16162.637906] sgpool-16                  8KB          8KB
[16162.638915] sgpool-8                   4KB          4KB
[16162.639923] mqueue_inode_cache         16KB         16KB
[16162.640952] kioctx                    31KB         31KB
[16162.641960] aio_kiocb                  7KB          7KB
[16162.642972] bio-248                    8KB          8KB
[16162.643986] posix_timers_cache         19KB         19KB
[16162.645008] UNIX                     321KB        330KB
[16162.646015] tcp_bind_bucket           16KB         16KB
[16162.647053] ip_fib_trie                7KB          7KB
[16162.648061] ip_fib_alias               7KB          7KB
[16162.649077] ip_dst_cache              56KB         72KB
[16162.650093] RAW                       15KB         15KB
[16162.651103] UDP                       45KB         45KB
[16162.652115] tw_sock_TCP               19KB         19KB
[16162.653128] request_sock_TCP           7KB          7KB
[16162.654142] TCP                       95KB         95KB
[16162.655161] hugetlbfs_inode_cache          7KB          7KB
[16162.656238] bio-288                   11KB         11KB
[16162.657248] eventpoll_pwq             23KB         23KB
[16162.658260] eventpoll_epi             68KB         68KB
[16162.659274] inotify_inode_mark          7KB          7KB
[16162.660298] request_queue             63KB         63KB
[16162.661311] blkdev_ioc                35KB         35KB
[16162.662324] bio-224                   76KB         76KB
[16162.663333] biovec-max               544KB        704KB
[16162.664343] biovec-128                64KB         64KB
[16162.665355] biovec-64                 48KB         48KB
[16162.666362] biovec-16                  8KB          8KB
[16162.667375] bio_integrity_payload          4KB          4KB
[16162.668458] khugepaged_mm_slot          7KB          7KB
[16162.669486] uid_cache                  7KB          7KB
[16162.670501] dmaengine-unmap-256         30KB         30KB
[16162.671569] dmaengine-unmap-128         15KB         15KB
[16162.672624] dmaengine-unmap-16          3KB          3KB
[16162.673673] dmaengine-unmap-2          4KB          4KB
[16162.674678] audit_buffer               7KB          7KB
[16162.675684] skbuff_fclone_cache         70KB         70KB
[16162.676723] skbuff_head_cache         28KB         32KB
[16162.677721] configfs_dir_cache          3KB          3KB
[16162.678735] file_lock_cache            7KB          7KB
[16162.679735] file_lock_ctx              7KB          7KB
[16162.680745] fsnotify_mark_connector          7KB          7KB
[16162.681840] net_namespace             31KB         31KB
[16162.682844] task_delay_info           87KB         87KB
[16162.683858] taskstats                 15KB         15KB
[16162.684863] proc_dir_entry           252KB        252KB
[16162.685861] pde_opener                 7KB          7KB
[16162.686894] seq_file                   8KB          8KB
[16162.687903] sigqueue                   7KB          7KB
[16162.688935] shmem_inode_cache        845KB        882KB
[16162.689942] kernfs_iattrs_cache         15KB         15KB
[16162.690984] kernfs_node_cache       4063KB       4107KB
[16162.691989] mnt_cache                252KB        252KB
[16162.693004] filp                     474KB        536KB
[16162.694011] names_cache               64KB         64KB
[16162.695015] avc_node                   7KB          7KB
[16162.696040] lsm_inode_cache         1102KB       1692KB
[16162.697057] lsm_file_cache           120KB        120KB
[16162.698063] key_jar                   28KB         28KB
[16162.699072] nsproxy                    7KB          7KB
[16162.700083] vm_area_struct          1271KB       1375KB
[16162.701092] mm_struct                135KB        157KB
[16162.702101] fs_cache                   8KB          8KB
[16162.703110] files_cache               94KB         94KB
[16162.704275] signal_cache             699KB        725KB
[16162.705290] sighand_cache           1206KB       1206KB
[16162.706300] task_struct             3580KB       3651KB
[16162.707316] cred_jar                 158KB        169KB
[16162.708337] anon_vma_chain           223KB        268KB
[16162.709348] anon_vma                 240KB        249KB
[16162.710360] pid                      184KB        184KB
[16162.711374] Acpi-Operand              74KB         74KB
[16162.712402] Acpi-ParseExt              3KB          3KB
[16162.713415] Acpi-Parse                 3KB          3KB
[16162.714422] Acpi-State                11KB         11KB
[16162.715455] Acpi-Namespace            15KB         15KB
[16162.716466] numa_policy                3KB          3KB
[16162.717475] trace_event_file         150KB        150KB
[16162.718483] ftrace_event_field        203KB        203KB
[16162.719512] pool_workqueue            12KB         12KB
[16162.720519] task_group                63KB         63KB
[16162.721526] vmap_area                388KB        392KB
[16162.722550] kmalloc-cg-4k             96KB         96KB
[16162.723562] kmalloc-cg-2k            160KB        160KB
[16162.724567] kmalloc-cg-1k            152KB        192KB
[16162.725582] kmalloc-cg-512            24KB         24KB
[16162.726585] kmalloc-cg-192            23KB         23KB
[16162.727604] kmalloc-cg-96              7KB          7KB
[16162.728609] kmalloc-cg-64              8KB          8KB
[16162.729619] kmalloc-cg-32              8KB          8KB
[16162.730633] kmalloc-cg-16              8KB          8KB
[16162.731635] kmalloc-cg-8               8KB          8KB
[16162.732648] kmalloc-8k              4760KB       4896KB
[16162.733652] kmalloc-4k             10980KB      11008KB
[16162.734662] kmalloc-2k              3424KB       3488KB
[16162.735681] kmalloc-1k             28142KB      28348KB
[16162.736808] kmalloc-512            14760KB      16316KB
[16162.737832] kmalloc-256              577KB        624KB
[16162.738877] kmalloc-192             2397KB       2795KB
[16162.739938] kmalloc-128              635KB        920KB
[16162.740952] kmalloc-96              2893KB       2909KB
[16162.741993] kmalloc-64              2310KB       2668KB
[16162.743025] kmalloc-32              1790KB       1816KB
[16162.744083] kmalloc-16               164KB        164KB
[16162.745099] kmalloc-8                 52KB         52KB
[16162.746110] kmem_cache_node           24KB         24KB
[16162.747117] kmem_cache               149KB        149KB
[16162.748125] Tasks state (memory values in pages):
[16162.749030] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[16162.750628] [    560]     0   560    30725        0   311296      530             0 systemd-journal
[16162.752299] [    598]     0   598    28690        0   217088      325         -1000 systemd-udevd
[16162.753946] [    733]    32   733    16800        0   172032      195             0 rpcbind
[16162.755492] [    737]     0   737    19253        0   155648      169         -1000 auditd
[16162.757030] [    765]    70   765    21366        0   176128      205             0 avahi-daemon
[16162.758652] [    766]    81   766    15815        0   176128      231          -900 dbus-daemon
[16162.760255] [    767]     0   767    31223        0   147456      185             0 irqbalance
[16162.761851] [    769]     0   769    54776        0   442368      536             0 sssd
[16162.763357] [    770]     0   770    97977        0   380928     1203             0 NetworkManager
[16162.765014] [    771]   998   771   407557        0   331776     1788             0 polkitd
[16162.766552] [    773]   995   773    37789        0   188416      196             0 chronyd
[16162.768111] [    775]    70   775    21302        0   151552      109             0 avahi-daemon
[16162.769733] [    790]     0   790    23087        1   192512      233         -1000 sshd
[16162.771239] [    791]     0   791   124104       58   446464     4701             0 tuned
[16162.772792] [    802]   994   802    31995        0   147456      184             0 munged
[16162.774338] [    804]     0   804    15005        0   151552      148             0 xinetd
[16162.775905] [    807]     0   807    57024        0   450560      809             0 sssd_be
[16162.777453] [    825]     0   825    97428        0   278528      510             0 rsyslogd
[16162.779027] [    838]    29   838    17942        0   176128     6297             0 rpc.statd
[16162.780607] [    859]     0   859    57317        0   471040      481             0 sssd_nss
[16162.782172] [    880]     0   880    25444        0   176128      175             0 gssproxy
[16162.783743] [    893]     0   893    34690        0   286720      366             0 cupsd
[16162.785290] [    897]     0   897    23157        0   208896      257             0 systemd-logind
[16162.786967] [   1045]     0  1045    30781        0   192512      255             0 master
[16162.788493] [   1057]    89  1057    37618        0   237568      274             0 qmgr
[16162.790001] [   1100]     0  1100   102309        0   294912      472             0 automount
[16162.791569] [   1106]     0  1106     9239        0   106496      242             0 crond
[16162.793101] [   1108]     0  1108    11001        0   110592       54             0 atd
[16162.794586] [   1126]     0  1126     3414        0    65536       33             0 agetty
[16162.796153] [   1127]     0  1127     4056        0    69632       37             0 agetty
[16162.797869] [ 279880]    89 279880    37605        0   233472      276             0 pickup
[16162.799424] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanityn test_56b - trevis-77vm3 crashed during sanityn test_56b

Issue Links

• is related to: LU-15941 sanity test_398b: timeouts with ZFS
• is related to: LU-18311 interop: sanity test_312: FAIL: blksz error, actual 4096, expected: 2 * 1 * 4096
• is related to: LU-9054 sanity test_312: FAIL: blksz error: , expected: 4096
• is related to: LU-13594 register OOM callback in Lustre
• is related to: LU-4865 osd-zfs: increase object block size dynamically as object grows
• is related to: LU-8137 fix llverdev to be able to write and verify large files in the filesystem