[LU-11122] NULL pointer dereference in fld_local_lookup()

• Type: Bug
• Resolution: Duplicate
• Priority: Blocker
• Labels: llnl
• Environment: lustre-2.8.2_2.chaos-1.ch6.x86_64 ; 4 MDTs used in DNE-1 fashion (remote dirs, no striped dirs) ; RHEL 7.5
• Severity: 1
• Assignee: Alex Zhuravlev
• Reporter: Olaf Faaland

Description

MDS nodes were power cycled during hardware maintenance. After they came back up, got below (some material redacted, see comments below for full console log contents):

BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
IP: [<ffffffffc0cdc392>] fld_local_lookup+0x52/0x270 [fld]
CPU: 17 PID: 180501 Comm: orph_cleanup_ls Kdump: loaded Tainted: P OE ------------ 3.10.0-862.3.2.1chaos.ch6.x86_64 #1
Call Trace:
[<ffffffffc06e8f6c>] ? dmu_tx_hold_object_impl+0x6c/0xc0 [zfs]
[<ffffffffc109ff28>] osd_fld_lookup+0x48/0xd0 [osd_zfs]
[<ffffffffc10a008a>] fid_is_on_ost+0xda/0x2f0 [osd_zfs]
[<ffffffffc10a02e9>] osd_get_name_n_idx+0x49/0xd00 [osd_zfs]
[<ffffffffc109902c>] ? osd_declare_attr_set+0x14c/0x730 [osd_zfs]
[<ffffffffc0753b7e>] ? zap_lookup_by_dnode+0x2e/0x30 [zfs]
[<ffffffffc1097510>] osd_declare_object_destroy+0xe0/0x3e0 [osd_zfs]
[<ffffffffc1139ffe>] lod_sub_object_declare_destroy+0xce/0x2d0 [lod]
[<ffffffffc1129700>] lod_declare_object_destroy+0x170/0x4a0 [lod]
[<ffffffffc1513689>] ? orph_declare_index_delete+0x179/0x460 [mdd]
[<ffffffffc1513f66>] orph_key_test_and_del+0x5f6/0xd30 [mdd]
[<ffffffffc1514c57>] __mdd_orphan_cleanup+0x5b7/0x840 [mdd]
[<ffffffffc15146a0>] ? orph_key_test_and_del+0xd30/0xd30 [mdd]
[<ffffffffbb2c05f1>] kthread+0xd1/0xe0
[<ffffffffbb2c0520>] ? insert_kthread_work+0x40/0x40
[<ffffffffbb9438b7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbb2c0520>] ? insert_kthread_work+0x40/0x40

Comments

[Peter Jones]

Alex

What do you recommend here?

Peter

[Peter Jones]

No problem. We'll discuss tomorrow whether further work could be useful to be able to avoid this kind of scenario.

[Olaf Faaland]

Thank you for responding so quickly.

[Olaf Faaland]

John, thanks for the suggestion.  I'll try it if this comes up again.

[Olaf Faaland]

The file system is back up.  This is no longer an emergency.

In the course of rebooting the nodes and mounting manually to get more information, the messages about orphan cleanup failing stopped appearing after recovery.  It's been about 6 minutes now since the MDTs completed recovery.

[Olaf Faaland]

Lustre only, no IML.

[Will Johnson]

Hi ofaaland,

Is this a Lustre only system or do you also have IML installed?

Regards,

Will

[John Hammond]

> Is there a way we can tell the MDT to skip orphan cleanup once, so it forgets about these orphaned objects? We don't really care if some unused objects are orphaned.

You could mount the backing FS and remove the files from the PENDING directory.

[Olaf Faaland]

Is there a way we can tell the MDT to skip orphan cleanup once, so it forgets about these orphaned objects?  We don't really care if some unused objects are orphaned.

[Olaf Faaland]

For our code, including the tags we build from, see the lustre-release-fe-llnl project in gerrit.

 

 

[Olaf Faaland]

Top of stack:

[2507417.726696] [<ffffffffc109ff28>] osd_fld_lookup+0x48/0xd0 [osd_zfs]
[2507417.726700] [<ffffffffc10a008a>] fid_is_on_ost+0xda/0x2f0 [osd_zfs]
[2507417.726703] [<ffffffffc10a02e9>] osd_get_name_n_idx+0x49/0xd00 [osd_zfs]

And point of NULL ptr dereference:

[2507417.726655] RIP: 0010:[<ffffffffc0cdc392>] [<ffffffffc0cdc392>] fld_local_lookup+0x52/0x270 [fld]

(gdb) l *(fld_local_lookup+0x52)
0x53c2 is in fld_local_lookup (/usr/src/debug/lustre-2.8.2_2.chaos/lustre/fld/fld_handler.c:218).
213 info = lu_context_key_get(&env->le_ctx, &fld_thread_key);
214 LASSERT(info != NULL);
215 erange = &info->fti_lrange;
216
217 /* Lookup it in the cache. */
218 rc = fld_cache_lookup(fld->lsf_cache, seq, erange);
219 if (rc == 0) {
220 if (unlikely(fld_range_type(erange) != fld_range_type(range) &&
221 !fld_range_is_any(range))) {
222 CERROR("%s: FLD cache range "DRANGE" does not match"

Variable info cannot be NULL, because it is local, not shared, and the ASSERT verified it was not NULL after it was last assigned. So it seems like line 218 really is where the NULL ptr deref is occurring:

218 rc = fld_cache_lookup(fld->lsf_cache, seq, erange);

fld was passed in and we have not check it for NULL, and that's the only pointer used. So it seems fld must be NULL.