[LU-10433] cfs_hash_destroy()) ASSERTION( !cfs_hash_with_assert_empty(hs) ) failed: hash lu_site_osd-zfs bucket 3(1) is not empty: 1 items left

Related — Sun, 18 Oct 2020 00:55:40 +0000

This seems to imply some sort of a cleanup problem for mdd-survey.

I hi this by doing make distclean in my build tree while sanity.sh was running and so it failed and then failed to cleanup I guess.

[446668.581081] Lustre: DEBUG MARKER: == sanity test 225a: Metadata survey sanity with zero-stripe ========================================= 00:03:52 (1514005432)
[446669.900084] LustreError: 13007:0:(echo_client.c:1795:echo_md_lookup()) lookup MDT0000-tests: rc = -2
[446669.902193] LustreError: 13007:0:(echo_client.c:2027:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2
[446823.607961] Lustre: DEBUG MARKER: == sanity test 225b: Metadata survey sanity with stripe_count = 1 ==================================== 00:06:27 (1514005587)
[446824.709130] LustreError: 31575:0:(echo_client.c:1795:echo_md_lookup()) lookup MDT0000-tests: rc = -2
[446824.731978] LustreError: 31575:0:(echo_client.c:1795:echo_md_lookup()) Skipped 1 previous similar message
[446824.735717] LustreError: 31575:0:(echo_client.c:2027:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2
[446824.740322] LustreError: 31575:0:(echo_client.c:2027:echo_md_destroy_internal()) Skipped 1 previous similar message
[447024.304815] Lustre: Unmounted lustre-client
[447024.427247] LustreError: 164-3: Can't find the device name (need mount option 'device=...')
[447024.428806] LustreError: 24705:0:(obd_mount.c:1419:lmd_parse()) Bad mount options user_xattr,flock
[447024.429745] LustreError: 24705:0:(obd_mount.c:1507:lustre_fill_super()) Unable to mount  (-22)
[447291.721845] sysctl (14834): drop_caches: 2
[447298.071029] sh (15312): drop_caches: 2
[447314.214683] LustreError: 16631:0:(mgs_handler.c:1582:mgs_device_fini()) header@ffff880257355040[0x0, 1, [0x200000003:0x0:0x0] hash exist]{

[447314.216987] LustreError: 16631:0:(mgs_handler.c:1582:mgs_device_fini()) ....local_storage@ffff880257355090

[447314.218709] LustreError: 16631:0:(mgs_handler.c:1582:mgs_device_fini()) ....osd-zfs@ffff88024ee18de8osd-zfs-object@ffff88024ee18de8

[447314.220805] LustreError: 16631:0:(mgs_handler.c:1582:mgs_device_fini()) } header@ffff880257355040

[447314.383840] LustreError: 16631:0:(osd_handler.c:1264:osd_device_free()) header@ffff880257355040[0x0, 1, [0x200000003:0x0:0x0] hash exist]{

[447314.385775] LustreError: 16631:0:(osd_handler.c:1264:osd_device_free()) ....local_storage@ffff880257355090

[447314.387236] LustreError: 16631:0:(osd_handler.c:1264:osd_device_free()) ....osd-zfs@ffff88024ee18de8osd-zfs-object@ffff88024ee18de8

[447314.388665] LustreError: 16631:0:(osd_handler.c:1264:osd_device_free()) } header@ffff880257355040

[447314.390150] LustreError: 16631:0:(hash.c:1111:cfs_hash_destroy()) ASSERTION( !cfs_hash_with_assert_empty(hs) ) failed: hash lu_site_osd-zfs bucket 3(1) is not  empty: 1 items left
[447314.399178] LustreError: 16631:0:(hash.c:1111:cfs_hash_destroy()) LBUG
[447314.414803] Pid: 16631, comm: umount
[447314.415504] 
Call Trace:
[447314.440545]  [<ffffffffa020a7ce>] libcfs_call_trace+0x4e/0x60 [libcfs]
[447314.441076]  [<ffffffffa020a85c>] lbug_with_loc+0x4c/0xb0 [libcfs]
[447314.441650]  [<ffffffffa021e3ac>] cfs_hash_putref+0x3ac/0x500 [libcfs]
[447314.442191]  [<ffffffffa03ef9b4>] lu_site_fini+0x54/0xa0 [obdclass]
[447314.442923]  [<ffffffffa119df5b>] osd_device_free+0x9b/0x240 [osd_zfs]
[447314.443558]  [<ffffffffa03c480c>] class_free_dev+0x44c/0x650 [obdclass]
[447314.444067]  [<ffffffffa03c4c30>] class_export_put+0x220/0x2f0 [obdclass]
[447314.446687]  [<ffffffffa03c6755>] class_unlink_export+0x135/0x170 [obdclass]
[447314.447218]  [<ffffffffa03dbca0>] class_decref+0x80/0x160 [obdclass]
[447314.447718]  [<ffffffffa03dc0f1>] class_detach+0x1b1/0x2e0 [obdclass]
[447314.448241]  [<ffffffffa03e2b89>] class_process_config+0x19d9/0x28a0 [obdclass]
[447314.449135]  [<ffffffff811cd4f9>] ? __kmalloc+0x649/0x660
[447314.449626]  [<ffffffffa0215ed7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[447314.450318]  [<ffffffffa03e3c30>] class_manual_cleanup+0x1e0/0x6d0 [obdclass]
[447314.451201]  [<ffffffffa119dbda>] osd_obd_disconnect+0x9a/0x190 [osd_zfs]
[447314.451722]  [<ffffffffa03ecf1b>] lustre_put_lsi+0x15b/0x530 [obdclass]
[447314.452243]  [<ffffffffa03ed878>] lustre_common_put_super+0x588/0xb50 [obdclass]
[447314.453273]  [<ffffffffa0411919>] server_put_super+0x119/0xca0 [obdclass]
[447314.453961]  [<ffffffff811efa46>] generic_shutdown_super+0x56/0xe0
[447314.454627]  [<ffffffff811efe22>] kill_anon_super+0x12/0x20
[447314.455303]  [<ffffffffa03e6352>] lustre_kill_super+0x32/0x50 [obdclass]
[447314.456062]  [<ffffffff811f0329>] deactivate_locked_super+0x49/0x60
[447314.456735]  [<ffffffff811f0926>] deactivate_super+0x46/0x60
[447314.457423]  [<ffffffff8120f115>] mntput_no_expire+0xc5/0x120
[447314.458103]  [<ffffffff8121029f>] SyS_umount+0x9f/0x3c0
[447314.458785]  [<ffffffff8170fc49>] system_call_fastpath+0x16/0x1b
[447314.468749] 
[447314.469326] Kernel panic - not syncing: LBUG

Whamcloud Community JIRA

[LU-10433] cfs_hash_destroy()) ASSERTION( !cfs_hash_with_assert_empty(hs) ) failed: hash lu_site_osd-zfs bucket 3(1) is not empty: 1 items left