[#LU-1110] MDS Oops in osd_xattr_get() during file open by FID

[LU-1110] MDS Oops in osd_xattr_get() during file open by FID Created: 16/Feb/12 Updated: 28/Nov/13 Resolved: 29/Mar/12
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.1.0
Fix Version/s:	Lustre 2.3.0

Type:

Bug

Priority:

Major

Reporter:

Alexandre Louvet

Assignee:

Lai Siyao

Resolution:

Fixed

Votes:

Labels:

mn1

Severity:

Rank (Obsolete):

4677

Description

MDS crashed solid/multiple-times before we were able to identify the concerned File/FID and fix the situation by unlink'ing it !!!

The panic stack looks like following :
======================================
BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8
IP: [<ffffffffa0b0f05d>] osd_xattr_get+0x7d/0x170 [osd_ldiskfs]
PGD 0
Oops: 0000 1 SMP
last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:86:00.1/host14/rport-14:0-0/target14:0:0/14:0:0:2/timeout
CPU 4
Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgc(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ipmi_devintf ipmi_si ipmi_msghandler nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc acpi_cpufreq freq_table rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_round_robin dm_multipath usbhid hid ghes i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ehci_hcd uhci_hcd ioatdma hed sg lpfc scsi_transport_fc scsi_tgt igb dca ext4 jbd2 sd_mod crc_t10dif ahci megaraid_sas dm_mod [last unloaded: microcode]

Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgc(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ipmi_devintf ipmi_si ipmi_msghandler nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc acpi_cpufreq freq_table rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_round_robin dm_multipath usbhid hid ghes i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ehci_hcd uhci_hcd ioatdma hed sg lpfc scsi_transport_fc scsi_tgt igb dca ext4 jbd2 sd_mod crc_t10dif ahci megaraid_sas dm_mod [last unloaded: microcode]
Pid: 11774, comm: mdt_58 Tainted: G ---------------- T 2.6.32-131.12.1.bl6.Bull.26.x86_64 #1 bullx super-node
RSP: 0018:ffff8804d21ef490 EFLAGS: 00010202
RIP: 0010:[<ffffffffa0b0f05d>] [<ffffffffa0b0f05d>] osd_xattr_get+0x7d/0x170 [osd_ldiskfs]
RSP: 0018:ffff8804d21ef490 EFLAGS: 00010202
RAX: ffff8808592058c0 RBX: ffff8804d21efe90 RCX: ffffffffa0a824e6
RDX: ffff88105b220dc0 RSI: ffffffffa0b1b1e0 RDI: ffff8804d21efe90
RBP: ffff8804d21ef4e0 R08: 0000000000000000 R09: ffffffffa0b17900
R10: ffff8808592058c0 R11: ffff8808592058d0 R12: ffff8808599d1300
R13: 0000000000000000 R14: ffff8808599d1300 R15: ffff8810598ca3a8
FS: 00007fd36d430700(0000) GS:ffff880044e40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000000f8 CR3: 00000018561a6000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt_58 (pid: 11774, threadinfo ffff8804d21ec000, task ffff8804d21e9740)
Stack:
ffff8808586e1800 ffffffffa0a824e6 ffff88105ad26000 0000000000000000
<0> ffff88087fc103c0 ffff8804d21efe90 ffff8808599d13c0 0000000000000000
<0> ffff8804d21ef64c ffff8808586e1800 ffff8804d21ef560 ffffffffa0a564e6
Call Trace:
[<ffffffffa0a564e6>] mdd_get_md+0x96/0x350 [mdd]
[<ffffffff8147d876>] ? down_read+0x16/0x30
[<ffffffffa0a567ec>] mdd_get_md_locked+0x4c/0x70 [mdd]
[<ffffffffa0a5a563>] mdd_lov_create+0xc43/0x21f0 [mdd]
[<ffffffffa0a64a2e>] mdd_create_data+0x37e/0x5c0 [mdd]
[<ffffffffa0a79926>] ? mdd_read_unlock+0x26/0x30 [mdd]
[<ffffffffa0b35056>] cml_create_data+0xb6/0x260 [cmm]
[<ffffffffa0b36c29>] ? cml_xattr_get+0x89/0x1d0 [cmm]
[<ffffffffa06a3190>] ? lustre_swab_mdt_body+0x0/0x150 [ptlrpc]
[<ffffffffa0ad0c22>] mdt_finish_open+0x13c2/0x18e0 [mdt]
[<ffffffffa0b3460f>] ? cml_attr_get+0x7f/0x1c0 [cmm]
[<ffffffffa0ad2b39>] mdt_reint_open+0x19f9/0x2c50 [mdt]
[<ffffffffa0a71ff6>] ? md_ucred+0x26/0x60 [mdd]
[<ffffffffa0a9e5f5>] ? mdt_ucred+0x15/0x20 [mdt]
[<ffffffffa0ab586f>] ? mdt_root_squash+0x2f/0x450 [mdt]
[<ffffffffa0abaabf>] mdt_reint_rec+0x3f/0x100 [mdt]
[<ffffffffa06a0b54>] ? lustre_msg_get_flags+0x34/0xa0 [ptlrpc]
[<ffffffffa0ab2f64>] mdt_reint_internal+0x6d4/0x9f0 [mdt]
[<ffffffffa0aa057e>] ? mdt_intent_fixup_resent+0x4e/0x270 [mdt]
[<ffffffffa0ab35e5>] mdt_intent_reint+0x245/0x600 [mdt]
[<ffffffffa04f4625>] ? cfs_hash_bd_lookup_intent+0xe5/0x130 [libcfs]
[<ffffffffa06a1f50>] ? lustre_swab_ldlm_intent+0x0/0x20 [ptlrpc]
[<ffffffffa0aab770>] mdt_intent_policy+0x3c0/0x6b0 [mdt]
[<ffffffff81042890>] ? fair_enqueue_task_fair+0x190/0x350
[<ffffffffa058c521>] ? class_handle_hash+0xa1/0x280 [obdclass]
[<ffffffffa0659afa>] ldlm_lock_enqueue+0x2da/0xa50 [ptlrpc]
[<ffffffffa0678305>] ? ldlm_export_lock_get+0x15/0x20 [ptlrpc]
[<ffffffffa04f3692>] ? cfs_hash_bd_add_locked+0x62/0x90 [libcfs]
[<ffffffffa0680227>] ldlm_handle_enqueue0+0x447/0x1090 [ptlrpc]
[<ffffffffa0aa6f81>] ? mdt_unpack_req_pack_rep+0x51/0x5d0 [mdt]
[<ffffffffa0aab2ea>] mdt_enqueue+0x4a/0x110 [mdt]
[<ffffffffa0aa7dd5>] mdt_handle_common+0x8d5/0x1810 [mdt]
[<ffffffffa069e2d4>] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc]
[<ffffffffa0aa8de5>] mdt_regular_handle+0x15/0x20 [mdt]
[<ffffffffa06af019>] ptlrpc_main+0xc79/0x19d0 [ptlrpc]
[<ffffffff810017bc>] ? __switch_to+0x1ac/0x320
[<ffffffffa06ae3a0>] ? ptlrpc_main+0x0/0x19d0 [ptlrpc]
[<ffffffff810041aa>] child_rip+0xa/0x20
[<ffffffffa06ae3a0>] ? ptlrpc_main+0x0/0x19d0 [ptlrpc]
[<ffffffffa06ae3a0>] ? ptlrpc_main+0x0/0x19d0 [ptlrpc]
[<ffffffff810041a0>] ? child_rip+0x0/0x20
Code: 45 c0 49 8b 04 24 f6 40 20 03 75 1f b9 f9 07 00 00 48 c7 c2 57 81 b1 a0 48 c7 c6 e8 8c b1 a0 48 c7 c7 75 87 b1 a0 e8 83 1d 9e ff <49> 8b 85 f8 00 00 00 48 85 c0 0f 84 84 00 00 00 48 83 b8 90 00
RIP [<ffffffffa0b0f05d>] osd_xattr_get+0x7d/0x170 [osd_ldiskfs]
RSP <ffff8804d21ef490>
CR2: 00000000000000f8
======================================

The reason of the Oops is because osd_xattr_get() references obj->oo_inode which is NULL.

This situation should be inherited from osd_fid_lookup() which returns 0 if osd_oi_lookup() returns ENOENT and does not initialize oo_inode ...

This comes from extensive "by-FID" direct operations used by customer's tools which may trigger some FileSystem inconsistencies (causing a FID not to be correctly resolved ??...) not beeing handled in "by-FID" access-method code.

And 1st question coming to my mind, is "by-FID" feature/access-method already available and safe for customer's usage ???

Comments

Comment by Peter Jones [ 16/Feb/12 ]

Lai

Could you please look into this one?

Thanks

Peter

Comment by Lai Siyao [ 27/Feb/12 ]

AFAIK, there is just an architecture design for open-by-fid feature, and security impact of this feature is not included in the design. As for the code, the support on MDT is incomplete, and no interface on client, that means no test is done for this. So basically this feature is not ready for customer usage.

Oleg, if I miss anything, please correct me.

Comment by Lai Siyao [ 27/Feb/12 ]

Ahh, I found the open-by-fid code is almost there, but lack of sanity test. I'll add some there to ensure normal operations won't crash.

Comment by Lai Siyao [ 29/Feb/12 ]

Review is on http://review.whamcloud.com/#change,2224.