Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11699

lod_device_free()) ASSERTION( atomic_read(&lu->ld_ref) == 0 ) failed: lu is ffff881a7e690000

Details

    • Bug
    • Resolution: Fixed
    • Minor
    • Lustre 2.13.0
    • None
    • 3
    • 9223372036854775807

    Description

      [597537.289392] LustreError: 14605:0:(lod_dev.c:1692:lod_device_free()) ASSERTION( atomic_read(&lu->ld_ref) == 0 ) failed: lu is ffff881a7e690000
      [597537.289393] LustreError: 14605:0:(lod_dev.c:1692:lod_device_free()) LBUG
      [597537.289393] Pid: 14605, comm: umount
      [597537.289394]
      Call Trace:
      [597537.289406] [<ffffffffc0a697ae>] libcfs_call_trace+0x4e/0x60 [libcfs]
      [597537.289410] [<ffffffffc0a6983c>] lbug_with_loc+0x4c/0xb0 [libcfs]
      [597537.289417] [<ffffffffc14dcfa3>] lod_device_free+0x2c3/0x300 [lod]
      [597537.289436] [<ffffffffc0bc9572>] class_free_dev+0x462/0x670 [obdclass]
      [597537.289446] [<ffffffffc0bc99a0>] class_export_put+0x220/0x2f0 [obdclass]
      [597537.289457] [<ffffffffc0bcb4b5>] class_unlink_export+0x135/0x170 [obdclass]
      [597537.289469] [<ffffffffc0be1550>] class_decref+0x80/0x160 [obdclass]
      [597537.289481] [<ffffffffc0be19b3>] class_detach+0x1b3/0x2e0 [obdclass]
      [597537.289493] [<ffffffffc0be85a7>] class_process_config+0x1987/0x2810 [obdclass]
      [597537.289495] [<ffffffff810c4c8d>] ? ttwu_do_activate.constprop.91+0x5d/0x70
      [597537.289501] [<ffffffffc0a76ef7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
      [597537.289513] [<ffffffffc0be9610>] class_manual_cleanup+0x1e0/0x710 [obdclass]
      [597537.289516] [<ffffffffc14dc9c6>] lod_obd_disconnect+0x86/0x1b0 [lod]
      [597537.289522] [<ffffffffc15bdd84>] mdd_process_config+0x404/0x650 [mdd]
      [597537.289531] [<ffffffffc13e9292>] mdt_stack_fini+0x2c2/0xd50 [mdt]
      [597537.289536] [<ffffffffc13ea0ab>] mdt_device_fini+0x38b/0x980 [mdt]
      [597537.289549] [<ffffffffc0be622a>] class_cleanup+0x8ea/0xc60 [obdclass]
      [597537.289560] [<ffffffffc0be727c>] class_process_config+0x65c/0x2810 [obdclass]
      [597537.289565] [<ffffffffc0a76ef7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
      [597537.289580] [<ffffffffc0be95f6>] class_manual_cleanup+0x1c6/0x710 [obdclass]
      [597537.289597] [<ffffffffc0c188ce>] server_put_super+0x8de/0xcd0 [obdclass]
      [597537.289600] [<ffffffff81208085>] generic_shutdown_super+0x75/0x100
      [597537.289601] [<ffffffff81208462>] kill_anon_super+0x12/0x20
      [597537.289616] [<ffffffffc0bec042>] lustre_kill_super+0x32/0x50 [obdclass]
      [597537.289618] [<ffffffff8120881e>] deactivate_locked_super+0x4e/0x70
      [597537.289619] [<ffffffff81208fa6>] deactivate_super+0x46/0x60
      [597537.289621] [<ffffffff8122655f>] cleanup_mnt+0x3f/0x80
      [597537.289622] [<ffffffff812265f2>] __cleanup_mnt+0x12/0x20
      [597537.289624] [<ffffffff810b087b>] task_work_run+0xbb/0xe0
      [597537.289626] [<ffffffff8102ab52>] do_notify_resume+0x92/0xb0
      [597537.289628] [<ffffffff816c1a5d>] int_signal+0x12/0x17
      [597537.289628]
      [597537.289629] Kernel panic - not syncing: LBUG
      [597537.289630] CPU: 4 PID: 14605 Comm: umount Tainted: G OE ------------ 3.10.0-693.21.1.x3.1.9.x86_64 #1
      [597537.289631] Hardware name: Seagate Laguna Seca/Laguna Seca, BIOS v02.0034 09/28/2017
      [597537.289631] Call Trace:
      [597537.289634] [<ffffffff816ae7c8>] dump_stack+0x19/0x1b
      [597537.289635] [<ffffffff816a8634>] panic+0xe8/0x21f
      [597537.289640] [<ffffffffc0a69854>] lbug_with_loc+0x64/0xb0 [libcfs]
      [597537.289645] [<ffffffffc14dcfa3>] lod_device_free+0x2c3/0x300 [lod]
      [597537.289659] [<ffffffffc0bc9572>] class_free_dev+0x462/0x670 [obdclass]
      [597537.289673] [<ffffffffc0bc99a0>] class_export_put+0x220/0x2f0 [obdclass]
      [597537.289688] [<ffffffffc0bcb4b5>] class_unlink_export+0x135/0x170 [obdclass]
      [597537.289703] [<ffffffffc0be1550>] class_decref+0x80/0x160 [obdclass]
      [597537.289718] [<ffffffffc0be19b3>] class_detach+0x1b3/0x2e0 [obdclass]
      [597537.289733] [<ffffffffc0be85a7>] class_process_config+0x1987/0x2810 [obdclass]
      [597537.289735] [<ffffffff810c4c8d>] ? ttwu_do_activate.constprop.91+0x5d/0x70
      [597537.289741] [<ffffffffc0a76ef7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
      [597537.289756] [<ffffffffc0be9610>] class_manual_cleanup+0x1e0/0x710 [obdclass]
      [597537.289762] [<ffffffffc14dc9c6>] lod_obd_disconnect+0x86/0x1b0 [lod]
      [597537.289766] [<ffffffffc15bdd84>] mdd_process_config+0x404/0x650 [mdd]
      [597537.289774] [<ffffffffc13e9292>] mdt_stack_fini+0x2c2/0xd50 [mdt]
      [597537.289782] [<ffffffffc13ea0ab>] mdt_device_fini+0x38b/0x980 [mdt]
      [597537.289798] [<ffffffffc0be622a>] class_cleanup+0x8ea/0xc60 [obdclass]
      [597537.289813] [<ffffffffc0be727c>] class_process_config+0x65c/0x2810 [obdclass]
      [597537.289820] [<ffffffffc0a76ef7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
      [597537.289835] [<ffffffffc0be95f6>] class_manual_cleanup+0x1c6/0x710 [obdclass]
      [597537.289868] [<ffffffffc0c188ce>] server_put_super+0x8de/0xcd0 [obdclass]
      [597537.289871] [<ffffffff81208085>] generic_shutdown_super+0x75/0x100
      [597537.289873] [<ffffffff81208462>] kill_anon_super+0x12/0x20
      [597537.289888] [<ffffffffc0bec042>] lustre_kill_super+0x32/0x50 [obdclass]
      [597537.289890] [<ffffffff8120881e>] deactivate_locked_super+0x4e/0x70
      [597537.289892] [<ffffffff81208fa6>] deactivate_super+0x46/0x60
      [597537.289894] [<ffffffff8122655f>] cleanup_mnt+0x3f/0x80
      [597537.289895] [<ffffffff812265f2>] __cleanup_mnt+0x12/0x20
      [597537.289897] [<ffffffff810b087b>] task_work_run+0xbb/0xe0
      [597537.289898] [<ffffffff8102ab52>] do_notify_resume+0x92/0xb0
      [597537.289900] [<ffffffff816c1a5d>] int_signal+0x12/0x17

      Attachments

        Issue Links

          Activity

            [LU-11699] lod_device_free()) ASSERTION( atomic_read(&lu->ld_ref) == 0 ) failed: lu is ffff881a7e690000
            pjones Peter Jones added a comment -

            Landed for 2.13

            pjones Peter Jones added a comment - Landed for 2.13

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33715/
            Subject: LU-11699 lfsck: Umount while running LFSCK
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 2d3d322546c14d7b35cfd3b160faa3e747a48315

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33715/ Subject: LU-11699 lfsck: Umount while running LFSCK Project: fs/lustre-release Branch: master Current Patch Set: Commit: 2d3d322546c14d7b35cfd3b160faa3e747a48315

            Andriy Skulysh (c17819@cray.com) uploaded a new patch: https://review.whamcloud.com/33715
            Subject: LU-11699 lfsck: Umount while running LFSCK
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: a15f8956b5fa9f07b85c21958a343a0cf1c8fdc9

            gerrit Gerrit Updater added a comment - Andriy Skulysh (c17819@cray.com) uploaded a new patch: https://review.whamcloud.com/33715 Subject: LU-11699 lfsck: Umount while running LFSCK Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: a15f8956b5fa9f07b85c21958a343a0cf1c8fdc9

            lfsck_assistant_engine is exiting but lfsck_thread_args_fini() wasn't called yet

            crash> bt 13988
            PID: 13988  TASK: ffff8811f1a01fa0  CPU: 8   COMMAND: "lfsck_layout"
             #0 [ffff881fff608e48] crash_nmi_callback at ffffffff810502b7
             #1 [ffff881fff608e58] nmi_handle at ffffffff816b992c
             #2 [ffff881fff608eb0] do_nmi at ffffffff816b9b5d
             #3 [ffff881fff608ef0] end_repeat_nmi at ffffffff816b8d89
                [exception RIP: delay_tsc+58]
                RIP: ffffffff81336c8a  RSP: ffff881a39e73988  RFLAGS: 00000046
                RAX: 000000007f231ad1  RBX: ffffffff81fe6f40  RCX: 0004abd47f2319d9
                RDX: 000000000004abd4  RSI: 0000000000000008  RDI: 0000000000000899
                RBP: ffff881a39e73988   R8: 0000000000000000   R9: 0000000000000000
                R10: 0000000000000000  R11: ffff881a39e738ae  R12: 0000000000002701
                R13: 0000000000000020  R14: ffffffff81d9e1f0  R15: 0000000000000090
                ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
            --- <NMI exception stack> ---
             #4 [ffff881a39e73988] delay_tsc at ffffffff81336c8a
             #5 [ffff881a39e73990] __const_udelay at ffffffff81336bcd
             #6 [ffff881a39e739a0] wait_for_xmitr at ffffffff8141a390
             #7 [ffff881a39e739c8] serial8250_console_putchar at ffffffff8141a58c
             #8 [ffff881a39e739e8] uart_console_write at ffffffff81414fac
             #9 [ffff881a39e73a18] serial8250_console_write at ffffffff8141a4cf
            #10 [ffff881a39e73a60] call_console_drivers.constprop.17 at ffffffff8108c0c3
            #11 [ffff881a39e73a88] console_unlock at ffffffff8108d868
            #12 [ffff881a39e73ac8] vprintk_emit at ffffffff8108dc74
            #13 [ffff881a39e73b38] vprintk_default at ffffffff8108dfe9
            #14 [ffff881a39e73b48] printk at ffffffff816a87cb
            #15 [ffff881a39e73ba8] cfs_print_to_console at ffffffffc0a6957a [libcfs]
            #16 [ffff881a39e73bd8] libcfs_debug_vmsg2 at ffffffffc0a768c4 [libcfs]
            #17 [ffff881a39e73d20] libcfs_debug_msg at ffffffffc0a76ef7 [libcfs]
            #18 [ffff881a39e73d80] lfsck_assistant_engine at ffffffffc1347b37 [lfsck]
            #19 [ffff881a39e73ec8] kthread at ffffffff810b4031
            #20 [ffff881a39e73f50] ret_from_fork at ffffffff816c155d
            
            askulysh Andriy Skulysh added a comment - lfsck_assistant_engine is exiting but lfsck_thread_args_fini() wasn't called yet crash> bt 13988 PID: 13988 TASK: ffff8811f1a01fa0 CPU: 8 COMMAND: "lfsck_layout" #0 [ffff881fff608e48] crash_nmi_callback at ffffffff810502b7 #1 [ffff881fff608e58] nmi_handle at ffffffff816b992c #2 [ffff881fff608eb0] do_nmi at ffffffff816b9b5d #3 [ffff881fff608ef0] end_repeat_nmi at ffffffff816b8d89 [exception RIP: delay_tsc+58] RIP: ffffffff81336c8a RSP: ffff881a39e73988 RFLAGS: 00000046 RAX: 000000007f231ad1 RBX: ffffffff81fe6f40 RCX: 0004abd47f2319d9 RDX: 000000000004abd4 RSI: 0000000000000008 RDI: 0000000000000899 RBP: ffff881a39e73988 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000000000 R11: ffff881a39e738ae R12: 0000000000002701 R13: 0000000000000020 R14: ffffffff81d9e1f0 R15: 0000000000000090 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- #4 [ffff881a39e73988] delay_tsc at ffffffff81336c8a #5 [ffff881a39e73990] __const_udelay at ffffffff81336bcd #6 [ffff881a39e739a0] wait_for_xmitr at ffffffff8141a390 #7 [ffff881a39e739c8] serial8250_console_putchar at ffffffff8141a58c #8 [ffff881a39e739e8] uart_console_write at ffffffff81414fac #9 [ffff881a39e73a18] serial8250_console_write at ffffffff8141a4cf #10 [ffff881a39e73a60] call_console_drivers.constprop.17 at ffffffff8108c0c3 #11 [ffff881a39e73a88] console_unlock at ffffffff8108d868 #12 [ffff881a39e73ac8] vprintk_emit at ffffffff8108dc74 #13 [ffff881a39e73b38] vprintk_default at ffffffff8108dfe9 #14 [ffff881a39e73b48] printk at ffffffff816a87cb #15 [ffff881a39e73ba8] cfs_print_to_console at ffffffffc0a6957a [libcfs] #16 [ffff881a39e73bd8] libcfs_debug_vmsg2 at ffffffffc0a768c4 [libcfs] #17 [ffff881a39e73d20] libcfs_debug_msg at ffffffffc0a76ef7 [libcfs] #18 [ffff881a39e73d80] lfsck_assistant_engine at ffffffffc1347b37 [lfsck] #19 [ffff881a39e73ec8] kthread at ffffffff810b4031 #20 [ffff881a39e73f50] ret_from_fork at ffffffff816c155d

            People

              askulysh Andriy Skulysh
              askulysh Andriy Skulysh
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: