Details
-
Bug
-
Resolution: Duplicate
-
Blocker
-
None
-
None
-
lustre-2.8.2_2.chaos-1.ch6.x86_64
4 MDTs used in DNE-1 fashion (remote dirs, no striped dirs)
RHEL 7.5
-
1
-
9223372036854775807
Description
MDS nodes were power cycled during hardware maintenance. After they came back up, got below (some material redacted, see comments below for full console log contents):
BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
IP: [<ffffffffc0cdc392>] fld_local_lookup+0x52/0x270 [fld]
CPU: 17 PID: 180501 Comm: orph_cleanup_ls Kdump: loaded Tainted: P OE ------------ 3.10.0-862.3.2.1chaos.ch6.x86_64 #1
Call Trace:
[<ffffffffc06e8f6c>] ? dmu_tx_hold_object_impl+0x6c/0xc0 [zfs]
[<ffffffffc109ff28>] osd_fld_lookup+0x48/0xd0 [osd_zfs]
[<ffffffffc10a008a>] fid_is_on_ost+0xda/0x2f0 [osd_zfs]
[<ffffffffc10a02e9>] osd_get_name_n_idx+0x49/0xd00 [osd_zfs]
[<ffffffffc109902c>] ? osd_declare_attr_set+0x14c/0x730 [osd_zfs]
[<ffffffffc0753b7e>] ? zap_lookup_by_dnode+0x2e/0x30 [zfs]
[<ffffffffc1097510>] osd_declare_object_destroy+0xe0/0x3e0 [osd_zfs]
[<ffffffffc1139ffe>] lod_sub_object_declare_destroy+0xce/0x2d0 [lod]
[<ffffffffc1129700>] lod_declare_object_destroy+0x170/0x4a0 [lod]
[<ffffffffc1513689>] ? orph_declare_index_delete+0x179/0x460 [mdd]
[<ffffffffc1513f66>] orph_key_test_and_del+0x5f6/0xd30 [mdd]
[<ffffffffc1514c57>] __mdd_orphan_cleanup+0x5b7/0x840 [mdd]
[<ffffffffc15146a0>] ? orph_key_test_and_del+0xd30/0xd30 [mdd]
[<ffffffffbb2c05f1>] kthread+0xd1/0xe0
[<ffffffffbb2c0520>] ? insert_kthread_work+0x40/0x40
[<ffffffffbb9438b7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbb2c0520>] ? insert_kthread_work+0x40/0x40
Attachments
Activity
Peter, can you backport this to 2.8fe? This way it gets some automated sanity tests run against it, which we aren't really set up to do.
Is any further action needed here or can this ticket be marked as a duplicate of LU-7206?
You're right, this was in the context of a umount. That all seems to fit.
in 2.8.2_2.chaos the sequence at umount will be:
mdt_fini() > mdt_fld_fini() ( ss>ss_server_fld = NULL ) -> mdt_stack_fini() -> ldo_process_config*LCFG_CLEANUP) -> mdd_device_shutdown() -> mdd_generic_thread_stop(mdd_orph_cleanup_thread)
IOW, ss_server_fld can be reset to NULL while the orphan thread is still running
was it possible that umount was initiated at that time? I see that 2.8.2_2.chaos stops orphan thread in mdd_shutdown() (LCFG_CLEANUP path) while master branch stops orphan thread in the preceding phase (LCFG_PRE_CLEANUP path)
Dupe of
LU-7206