[LU-9808] recovery-small test 102: osd_destroy()) ASSERTION( !lu_object_is_dying(dt->do_lu.lo_header) ) failed

Related — Fri, 12 Aug 2022 22:01:10 +0000
master + test-only change that enabled 2 recovery-small tests: https://review.whamcloud.com/#/c/27382/1
causes this very consistent crash for me in recovery-small test 102, that is somewhat similar to ~~LU-8060~~:
[ 4069.911603] Lustre: DEBUG MARKER: == recovery-small test 102: IR: New client gets updated nidtbl after MGS restart ===================== 18:41:40 (1501368100)
[ 4074.881029] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4074.884748] LustreError: Skipped 1 previous similar message
[ 4083.127141] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5
[ 4083.133413] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc
[ 4083.482673] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[ 4085.776322] Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid
[ 4088.448773] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16965 to 0x0:17057
[ 4089.173897] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec
[ 4096.630036] Lustre: Unmounted lustre-client
[ 4097.292327] LustreError: 32347:0:(osd_handler.c:3197:osd_destroy()) ASSERTION( !lu_object_is_dying(dt->do_lu.lo_header) ) failed: 
[ 4097.293334] LustreError: 32347:0:(osd_handler.c:3197:osd_destroy()) LBUG
[ 4097.294142] Pid: 32347, comm: umount
[ 4097.294810] 
Call Trace:
[ 4097.295666]  [<ffffffffa02207ce>] libcfs_call_trace+0x4e/0x60 [libcfs]
[ 4097.296291]  [<ffffffffa022085c>] lbug_with_loc+0x4c/0xb0 [libcfs]
[ 4097.296861]  [<ffffffffa0addc1e>] osd_destroy+0x50e/0x740 [osd_ldiskfs]
[ 4097.297366]  [<ffffffffa0adcbbf>] ? osd_ref_del+0x13f/0x690 [osd_ldiskfs]
[ 4097.298166]  [<ffffffffa0345397>] llog_osd_destroy+0x347/0x890 [obdclass]
[ 4097.298809]  [<ffffffffa0330608>] llog_destroy+0x2f8/0x3d0 [obdclass]
[ 4097.299339]  [<ffffffffa0338b9b>] llog_cat_close+0xfb/0x230 [obdclass]
[ 4097.300058]  [<ffffffffa0b9c584>] mdd_changelog_fini+0x64/0x1d0 [mdd]
[ 4097.300538]  [<ffffffffa0b9dc8a>] mdd_process_config+0x16a/0x600 [mdd]
[ 4097.301169]  [<ffffffffa0bfaa5c>] mdt_stack_fini+0x2bc/0xcf0 [mdt]
[ 4097.301824]  [<ffffffffa0bfbccf>] mdt_device_fini+0x83f/0xfb0 [mdt]
[ 4097.302359]  [<ffffffffa0368b64>] class_cleanup+0x7b4/0xcf0 [obdclass]
[ 4097.303037]  [<ffffffffa036b0dd>] class_process_config+0x19cd/0x23b0 [obdclass]
[ 4097.303888]  [<ffffffff811cd4f9>] ? __kmalloc+0x649/0x660
[ 4097.304378]  [<ffffffffa022bcc7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[ 4097.305199]  [<ffffffffa036bc86>] class_manual_cleanup+0x1c6/0x6d0 [obdclass]
[ 4097.305723]  [<ffffffffa03994de>] server_put_super+0x8ae/0xca0 [obdclass]
[ 4097.306233]  [<ffffffff811efa46>] generic_shutdown_super+0x56/0xe0
[ 4097.306897]  [<ffffffff811efe22>] kill_anon_super+0x12/0x20
[ 4097.307404]  [<ffffffffa036e3c2>] lustre_kill_super+0x32/0x50 [obdclass]
[ 4097.317922]  [<ffffffff811f0329>] deactivate_locked_super+0x49/0x60
[ 4097.318539]  [<ffffffff811f0926>] deactivate_super+0x46/0x60
[ 4097.319040]  [<ffffffff8120f115>] mntput_no_expire+0xc5/0x120
[ 4097.319801]  [<ffffffff8121029f>] SyS_umount+0x9f/0x3c0
[ 4097.320293]  [<ffffffff8170fc49>] system_call_fastpath+0x16/0x1b
[ 4097.322496] 
[ 4097.323292] Kernel panic - not syncing: LBUG
[ 4097.323791] CPU: 13 PID: 32347 Comm: umount Tainted: P           OE  ------------   3.10.0-debug #2
[ 4097.324792] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 4097.325318]  ffffffffa023fed2 0000000014c35ea5 ffff8800986fb930 ffffffff816fd3e4
[ 4097.326384]  ffff8800986fb9b0 ffffffff816f8c34 ffffffff00000008 ffff8800986fb9c0
[ 4097.327395]  ffff8800986fb960 0000000014c35ea5 0000000014c35ea5 ffff88033e5ad948
[ 4097.328526] Call Trace:
[ 4097.329037]  [<ffffffff816fd3e4>] dump_stack+0x19/0x1b
[ 4097.329501]  [<ffffffff816f8c34>] panic+0xd8/0x1e7
[ 4097.330089]  [<ffffffffa0220874>] lbug_with_loc+0x64/0xb0 [libcfs]
[ 4097.330668]  [<ffffffffa0addc1e>] osd_destroy+0x50e/0x740 [osd_ldiskfs]
[ 4097.331221]  [<ffffffffa0adcbbf>] ? osd_ref_del+0x13f/0x690 [osd_ldiskfs]
[ 4097.331790]  [<ffffffffa0345397>] llog_osd_destroy+0x347/0x890 [obdclass]
[ 4097.332336]  [<ffffffffa0330608>] llog_destroy+0x2f8/0x3d0 [obdclass]
[ 4097.333068]  [<ffffffffa0338b9b>] llog_cat_close+0xfb/0x230 [obdclass]
[ 4097.333580]  [<ffffffffa0b9c584>] mdd_changelog_fini+0x64/0x1d0 [mdd]
[ 4097.334171]  [<ffffffffa0b9dc8a>] mdd_process_config+0x16a/0x600 [mdd]
[ 4097.334709]  [<ffffffffa0bfaa5c>] mdt_stack_fini+0x2bc/0xcf0 [mdt]
[ 4097.335240]  [<ffffffffa0bfbccf>] mdt_device_fini+0x83f/0xfb0 [mdt]
[ 4097.335775]  [<ffffffffa0368b64>] class_cleanup+0x7b4/0xcf0 [obdclass]
[ 4097.336539]  [<ffffffffa036b0dd>] class_process_config+0x19cd/0x23b0 [obdclass]
[ 4097.337676]  [<ffffffff811cd4f9>] ? __kmalloc+0x649/0x660
[ 4097.338462]  [<ffffffffa022bcc7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[ 4097.339263]  [<ffffffffa036bc86>] class_manual_cleanup+0x1c6/0x6d0 [obdclass]
[ 4097.339815]  [<ffffffffa03994de>] server_put_super+0x8ae/0xca0 [obdclass]
[ 4097.340591]  [<ffffffff811efa46>] generic_shutdown_super+0x56/0xe0
[ 4097.341146]  [<ffffffff811efe22>] kill_anon_super+0x12/0x20
[ 4097.341889]  [<ffffffffa036e3c2>] lustre_kill_super+0x32/0x50 [obdclass]
[ 4097.342685]  [<ffffffff811f0329>] deactivate_locked_super+0x49/0x60
[ 4097.343460]  [<ffffffff811f0926>] deactivate_super+0x46/0x60
[ 4097.344173]  [<ffffffff8120f115>] mntput_no_expire+0xc5/0x120
[ 4097.344881]  [<ffffffff8121029f>] SyS_umount+0x9f/0x3c0
[ 4097.345686]  [<ffffffff8170fc49>] system_call_fastpath+0x16/0x1b
Whamcloud Community JIRA

[LU-9808] recovery-small test 102: osd_destroy()) ASSERTION( !lu_object_is_dying(dt->do_lu.lo_header) ) failed
