Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20535

sanityn test_71b: Kernel panic - not syncing: Out of memory

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      This issue was created by maloo for Artem Blagodarenko <artem.blagodarenko@gmail.com>

      This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/8990c8e5-4845-442e-a74a-460e21df0f2a

      test_71b failed with the following error:

      onyx-143vm27 crashed during sanityn test_71b
      

      Test session details:
      clients: https://build.whamcloud.com/job/lustre-reviews/128239 - 5.14.0-611.55.1.el9_7.x86_64
      servers: https://build.whamcloud.com/job/lustre-reviews/128239 - 5.14.0-611.55.1.el9_7_lustre.x86_64

      [13497.079226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  SKIP: sanityn test_71b ORI-366\/LU-1941: FIEMAP unimplemented on ZFS
      [13497.363403] obd_memory max: 181029659, obd_memory current: 149009207
      [13497.364245] obd_memory max: 181029659, obd_memory current: 149009207
      [13497.521262] obd_memory max: 181029659, obd_memory current: 149009207
      [13497.523431] obd_memory max: 181029659, obd_memory current: 149009207
      [13497.540405] obd_memory max: 181029659, obd_memory current: 149009207
      [13497.543181] obd_memory max: 181029659, obd_memory current: 149009207
      [13497.543261] ll_ost00_026 invoked oom-killer: gfp_mask=0x42cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP), order=0, oom_score_adj=0
      [13497.543390] CPU: 1 PID: 34847 Comm: ll_ost00_026 Kdump: loaded Tainted: P           OE      ------  ---  5.14.0-611.55.1.el9_7_lustre.x86_64 #1
      [13497.543501] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
      [13497.543594] Call Trace:
      [13497.543692]  <TASK>
      [13497.543794]  dump_stack_lvl+0x34/0x48
      [13497.543929]  dump_header+0x49/0x212
      [13497.544039]  out_of_memory.cold+0x39/0x7e
      [13497.544141]  __alloc_pages_slowpath.constprop.0+0x793/0xb10
      [13497.544254]  __alloc_pages+0x21c/0x250
      [13497.544350]  alloc_pages_mpol+0x93/0x1e0
      [13497.544439]  allocate_slab+0x350/0x490
      [13497.544537]  ? get_any_partial+0x6b/0x190
      [13497.544630]  ___slab_alloc+0x33c/0x7c0
      [13497.544724]  ? __slab_free+0xcb/0x310
      [13497.544824]  ? spl_kmem_cache_alloc+0xcf/0x150 [spl]
      [13497.544945]  ? spl_kmem_cache_alloc+0xcf/0x150 [spl]
      [13497.545052]  kmem_cache_alloc+0x2d4/0x310
      [13497.545146]  spl_kmem_cache_alloc+0xcf/0x150 [spl]
      [13497.545254]  dbuf_create+0x3d/0x2e0 [zfs]
      [13497.545872]  dbuf_hold_impl+0x3b5/0x6e0 [zfs]
      [13497.546137]  dbuf_hold_level+0x2b/0x60 [zfs]
      [13497.546392]  dnode_free_range+0x557/0x6e0 [zfs]
      [13497.546679]  dmu_free_long_range_impl+0x389/0x480 [zfs]
      [13497.546964]  dmu_free_long_range+0x72/0xb0 [zfs]
      [13497.547238]  osd_unlinked_object_free+0x44/0x3b0 [osd_zfs]
      [13497.547378]  osd_unlinked_list_emptify+0xaa/0xb0 [osd_zfs]
      [13497.547489]  osd_trans_stop+0x3bf/0x520 [osd_zfs]
      [13497.547596]  ofd_destroy+0x2fb/0x690 [ofd]
      [13497.547717]  ofd_destroy_by_fid+0x2af/0x440 [ofd]
      [13497.547839]  ofd_destroy_hdl+0x173/0x9c0 [ofd]
      [13497.547952]  tgt_handle_request0+0x149/0x770 [ptlrpc]
      [13497.548373]  tgt_request_handle+0x3fd/0xd00 [ptlrpc]
      [13497.548594]  ptlrpc_server_handle_request.isra.0+0x2e8/0xd80 [ptlrpc]
      [13497.548798]  ptlrpc_main+0x9be/0xea0 [ptlrpc]
      [13497.549109]  ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
      [13497.549415]  kthread+0x104/0x110
      [13497.549543]  ? __pfx_kthread+0x10/0x10
      [13497.549642]  ret_from_fork+0x2b/0x50
      [13497.549751]  </TASK>
      [13497.549861] Mem-Info:
      [13497.549956] active_anon:0 inactive_anon:19 isolated_anon:0
       active_file:8 inactive_file:31 isolated_file:0
       unevictable:0 dirty:9 writeback:0
       slab_reclaimable:32148 slab_unreclaimable:63771
       mapped:3 shmem:7 pagetables:462
       sec_pagetables:0 bounce:0
       kernel_misc_reclaimable:0
       free:17744 free_pcp:118 free_cma:0
      

      and then OOM

      [13497.566736] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
      [13497.839001] CPU: 1 PID: 34847 Comm: ll_ost00_026 Kdump: loaded Tainted: P           OE      ------  ---  5.14.0-611.55.1.el9_7_lustre.x86_64 #1
      [13497.841192] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
      [13497.842356] Call Trace:
      [13497.842788]  <TASK>
      [13497.843168]  dump_stack_lvl+0x34/0x48
      [13497.843807]  panic+0x107/0x2bb
      [13497.844370]  out_of_memory.cold+0x5e/0x7e
      [13497.845074]  __alloc_pages_slowpath.constprop.0+0x793/0xb10
      [13497.846039]  __alloc_pages+0x21c/0x250
      [13497.846690]  alloc_pages_mpol+0x93/0x1e0
      [13497.847380]  allocate_slab+0x350/0x490
      [13497.848039]  ? get_any_partial+0x6b/0x190
      [13497.848732]  ___slab_alloc+0x33c/0x7c0
      [13497.849385]  ? __slab_free+0xcb/0x310
      [13497.850022]  ? spl_kmem_cache_alloc+0xcf/0x150 [spl]
      [13497.850896]  ? spl_kmem_cache_alloc+0xcf/0x150 [spl]
      [13497.851754]  kmem_cache_alloc+0x2d4/0x310
      [13497.852454]  spl_kmem_cache_alloc+0xcf/0x150 [spl]
      [13497.853287]  dbuf_create+0x3d/0x2e0 [zfs]
      [13497.854214]  dbuf_hold_impl+0x3b5/0x6e0 [zfs]
      [13497.855141]  dbuf_hold_level+0x2b/0x60 [zfs]
      [13497.856055]  dnode_free_range+0x557/0x6e0 [zfs]
      [13497.857022]  dmu_free_long_range_impl+0x389/0x480 [zfs]
      [13497.858146]  dmu_free_long_range+0x72/0xb0 [zfs]
      [13497.859134]  osd_unlinked_object_free+0x44/0x3b0 [osd_zfs]
      [13497.860114]  osd_unlinked_list_emptify+0xaa/0xb0 [osd_zfs]
      [13497.861074]  osd_trans_stop+0x3bf/0x520 [osd_zfs]
      [13497.861906]  ofd_destroy+0x2fb/0x690 [ofd]
      [13497.862643]  ofd_destroy_by_fid+0x2af/0x440 [ofd]
      [13497.863478]  ofd_destroy_hdl+0x173/0x9c0 [ofd]
      [13497.864270]  tgt_handle_request0+0x149/0x770 [ptlrpc]
      [13497.865320]  tgt_request_handle+0x3fd/0xd00 [ptlrpc]
      [13497.866306]  ptlrpc_server_handle_request.isra.0+0x2e8/0xd80 [ptlrpc]
      [13497.867512]  ptlrpc_main+0x9be/0xea0 [ptlrpc]
      [13497.868369]  ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
      [13497.869311]  kthread+0x104/0x110
      [13497.869878]  ? __pfx_kthread+0x10/0x10
      [13497.870522]  ret_from_fork+0x2b/0x50
      [13497.871145]  </TASK>
      (END)
      

      VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
      sanityn test_71b - onyx-143vm27 crashed during sanityn test_71b

            wc-triage WC Triage
            maloo Maloo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: