[LU-11699] lod_device_free()) ASSERTION( atomic_read(&lu->ld_ref) == 0 ) failed: lu is ffff881a7e690000

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.13.0
• Type: Bug
• Priority: Minor
• Reporter: Andriy Skulysh
• Assignee: Andriy Skulysh
• Resolution: Fixed
• Labels: patch
• Severity: 3

Description

[597537.289392] LustreError: 14605:0:(lod_dev.c:1692:lod_device_free()) ASSERTION( atomic_read(&lu->ld_ref) == 0 ) failed: lu is ffff881a7e690000
[597537.289393] LustreError: 14605:0:(lod_dev.c:1692:lod_device_free()) LBUG
[597537.289393] Pid: 14605, comm: umount
[597537.289394]
Call Trace:
[597537.289406] [<ffffffffc0a697ae>] libcfs_call_trace+0x4e/0x60 [libcfs]
[597537.289410] [<ffffffffc0a6983c>] lbug_with_loc+0x4c/0xb0 [libcfs]
[597537.289417] [<ffffffffc14dcfa3>] lod_device_free+0x2c3/0x300 [lod]
[597537.289436] [<ffffffffc0bc9572>] class_free_dev+0x462/0x670 [obdclass]
[597537.289446] [<ffffffffc0bc99a0>] class_export_put+0x220/0x2f0 [obdclass]
[597537.289457] [<ffffffffc0bcb4b5>] class_unlink_export+0x135/0x170 [obdclass]
[597537.289469] [<ffffffffc0be1550>] class_decref+0x80/0x160 [obdclass]
[597537.289481] [<ffffffffc0be19b3>] class_detach+0x1b3/0x2e0 [obdclass]
[597537.289493] [<ffffffffc0be85a7>] class_process_config+0x1987/0x2810 [obdclass]
[597537.289495] [<ffffffff810c4c8d>] ? ttwu_do_activate.constprop.91+0x5d/0x70
[597537.289501] [<ffffffffc0a76ef7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[597537.289513] [<ffffffffc0be9610>] class_manual_cleanup+0x1e0/0x710 [obdclass]
[597537.289516] [<ffffffffc14dc9c6>] lod_obd_disconnect+0x86/0x1b0 [lod]
[597537.289522] [<ffffffffc15bdd84>] mdd_process_config+0x404/0x650 [mdd]
[597537.289531] [<ffffffffc13e9292>] mdt_stack_fini+0x2c2/0xd50 [mdt]
[597537.289536] [<ffffffffc13ea0ab>] mdt_device_fini+0x38b/0x980 [mdt]
[597537.289549] [<ffffffffc0be622a>] class_cleanup+0x8ea/0xc60 [obdclass]
[597537.289560] [<ffffffffc0be727c>] class_process_config+0x65c/0x2810 [obdclass]
[597537.289565] [<ffffffffc0a76ef7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[597537.289580] [<ffffffffc0be95f6>] class_manual_cleanup+0x1c6/0x710 [obdclass]
[597537.289597] [<ffffffffc0c188ce>] server_put_super+0x8de/0xcd0 [obdclass]
[597537.289600] [<ffffffff81208085>] generic_shutdown_super+0x75/0x100
[597537.289601] [<ffffffff81208462>] kill_anon_super+0x12/0x20
[597537.289616] [<ffffffffc0bec042>] lustre_kill_super+0x32/0x50 [obdclass]
[597537.289618] [<ffffffff8120881e>] deactivate_locked_super+0x4e/0x70
[597537.289619] [<ffffffff81208fa6>] deactivate_super+0x46/0x60
[597537.289621] [<ffffffff8122655f>] cleanup_mnt+0x3f/0x80
[597537.289622] [<ffffffff812265f2>] __cleanup_mnt+0x12/0x20
[597537.289624] [<ffffffff810b087b>] task_work_run+0xbb/0xe0
[597537.289626] [<ffffffff8102ab52>] do_notify_resume+0x92/0xb0
[597537.289628] [<ffffffff816c1a5d>] int_signal+0x12/0x17
[597537.289628]
[597537.289629] Kernel panic - not syncing: LBUG
[597537.289630] CPU: 4 PID: 14605 Comm: umount Tainted: G OE ------------ 3.10.0-693.21.1.x3.1.9.x86_64 #1
[597537.289631] Hardware name: Seagate Laguna Seca/Laguna Seca, BIOS v02.0034 09/28/2017
[597537.289631] Call Trace:
[597537.289634] [<ffffffff816ae7c8>] dump_stack+0x19/0x1b
[597537.289635] [<ffffffff816a8634>] panic+0xe8/0x21f
[597537.289640] [<ffffffffc0a69854>] lbug_with_loc+0x64/0xb0 [libcfs]
[597537.289645] [<ffffffffc14dcfa3>] lod_device_free+0x2c3/0x300 [lod]
[597537.289659] [<ffffffffc0bc9572>] class_free_dev+0x462/0x670 [obdclass]
[597537.289673] [<ffffffffc0bc99a0>] class_export_put+0x220/0x2f0 [obdclass]
[597537.289688] [<ffffffffc0bcb4b5>] class_unlink_export+0x135/0x170 [obdclass]
[597537.289703] [<ffffffffc0be1550>] class_decref+0x80/0x160 [obdclass]
[597537.289718] [<ffffffffc0be19b3>] class_detach+0x1b3/0x2e0 [obdclass]
[597537.289733] [<ffffffffc0be85a7>] class_process_config+0x1987/0x2810 [obdclass]
[597537.289735] [<ffffffff810c4c8d>] ? ttwu_do_activate.constprop.91+0x5d/0x70
[597537.289741] [<ffffffffc0a76ef7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[597537.289756] [<ffffffffc0be9610>] class_manual_cleanup+0x1e0/0x710 [obdclass]
[597537.289762] [<ffffffffc14dc9c6>] lod_obd_disconnect+0x86/0x1b0 [lod]
[597537.289766] [<ffffffffc15bdd84>] mdd_process_config+0x404/0x650 [mdd]
[597537.289774] [<ffffffffc13e9292>] mdt_stack_fini+0x2c2/0xd50 [mdt]
[597537.289782] [<ffffffffc13ea0ab>] mdt_device_fini+0x38b/0x980 [mdt]
[597537.289798] [<ffffffffc0be622a>] class_cleanup+0x8ea/0xc60 [obdclass]
[597537.289813] [<ffffffffc0be727c>] class_process_config+0x65c/0x2810 [obdclass]
[597537.289820] [<ffffffffc0a76ef7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[597537.289835] [<ffffffffc0be95f6>] class_manual_cleanup+0x1c6/0x710 [obdclass]
[597537.289868] [<ffffffffc0c188ce>] server_put_super+0x8de/0xcd0 [obdclass]
[597537.289871] [<ffffffff81208085>] generic_shutdown_super+0x75/0x100
[597537.289873] [<ffffffff81208462>] kill_anon_super+0x12/0x20
[597537.289888] [<ffffffffc0bec042>] lustre_kill_super+0x32/0x50 [obdclass]
[597537.289890] [<ffffffff8120881e>] deactivate_locked_super+0x4e/0x70
[597537.289892] [<ffffffff81208fa6>] deactivate_super+0x46/0x60
[597537.289894] [<ffffffff8122655f>] cleanup_mnt+0x3f/0x80
[597537.289895] [<ffffffff812265f2>] __cleanup_mnt+0x12/0x20
[597537.289897] [<ffffffff810b087b>] task_work_run+0xbb/0xe0
[597537.289898] [<ffffffff8102ab52>] do_notify_resume+0x92/0xb0
[597537.289900] [<ffffffff816c1a5d>] int_signal+0x12/0x17

Comments

[Andriy Skulysh]

lfsck_assistant_engine is exiting but lfsck_thread_args_fini() wasn't called yet

crash> bt 13988
PID: 13988  TASK: ffff8811f1a01fa0  CPU: 8   COMMAND: "lfsck_layout"
 #0 [ffff881fff608e48] crash_nmi_callback at ffffffff810502b7
 #1 [ffff881fff608e58] nmi_handle at ffffffff816b992c
 #2 [ffff881fff608eb0] do_nmi at ffffffff816b9b5d
 #3 [ffff881fff608ef0] end_repeat_nmi at ffffffff816b8d89
    [exception RIP: delay_tsc+58]
    RIP: ffffffff81336c8a  RSP: ffff881a39e73988  RFLAGS: 00000046
    RAX: 000000007f231ad1  RBX: ffffffff81fe6f40  RCX: 0004abd47f2319d9
    RDX: 000000000004abd4  RSI: 0000000000000008  RDI: 0000000000000899
    RBP: ffff881a39e73988   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: ffff881a39e738ae  R12: 0000000000002701
    R13: 0000000000000020  R14: ffffffff81d9e1f0  R15: 0000000000000090
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
 #4 [ffff881a39e73988] delay_tsc at ffffffff81336c8a
 #5 [ffff881a39e73990] __const_udelay at ffffffff81336bcd
 #6 [ffff881a39e739a0] wait_for_xmitr at ffffffff8141a390
 #7 [ffff881a39e739c8] serial8250_console_putchar at ffffffff8141a58c
 #8 [ffff881a39e739e8] uart_console_write at ffffffff81414fac
 #9 [ffff881a39e73a18] serial8250_console_write at ffffffff8141a4cf
#10 [ffff881a39e73a60] call_console_drivers.constprop.17 at ffffffff8108c0c3
#11 [ffff881a39e73a88] console_unlock at ffffffff8108d868
#12 [ffff881a39e73ac8] vprintk_emit at ffffffff8108dc74
#13 [ffff881a39e73b38] vprintk_default at ffffffff8108dfe9
#14 [ffff881a39e73b48] printk at ffffffff816a87cb
#15 [ffff881a39e73ba8] cfs_print_to_console at ffffffffc0a6957a [libcfs]
#16 [ffff881a39e73bd8] libcfs_debug_vmsg2 at ffffffffc0a768c4 [libcfs]
#17 [ffff881a39e73d20] libcfs_debug_msg at ffffffffc0a76ef7 [libcfs]
#18 [ffff881a39e73d80] lfsck_assistant_engine at ffffffffc1347b37 [lfsck]
#19 [ffff881a39e73ec8] kthread at ffffffff810b4031
#20 [ffff881a39e73f50] ret_from_fork at ffffffff816c155d

[Gerrit Updater]

Andriy Skulysh ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33715
Subject: LU-11699 lfsck: Umount while running LFSCK
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a15f8956b5fa9f07b85c21958a343a0cf1c8fdc9

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33715/
Subject: LU-11699 lfsck: Umount while running LFSCK
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2d3d322546c14d7b35cfd3b160faa3e747a48315

[Peter Jones]

Landed for 2.13