https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12615 Lustre <p>In the latest version of lustre file system, the mdt module has a null pointer dereference bug due to the lack of validation for specific fields of packets sent by client.</p> <p>The kernel panic:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [93830.742518] BUG: unable to handle kernel NULL pointer dereference at 0000000000000050 [93830.745373] IP: [&lt;ffffffffc101cc2a&gt;] mdt_object_lock_internal+0x4a/0x360 [mdt] [93830.748121] PGD 0 [93830.750311] Oops: 0000 [#1] SMP [93830.752631] Modules linked in: ofd(OE) ost(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) loop lustre(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) crc_t10dif crct10dif_generic ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper parport_pc cryptd parport virtio_balloon joydev i2c_piix4 pcspkr ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_net virtio_console virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul drm crct10dif_common crc32c_intel libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy [last unloaded: stap_f794b15c5a4eaa0a0f39cf777c459f09_5532] [93830.772472] [93830.774428] CPU: 0 PID: 5222 Comm: mdt00_004 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.10.1.el7_lustre.x86_64 #1 [93830.779447] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 3288b3c 04/01/2014 [93830.782131] task: ffff9e93fb009040 ti: ffff9e90369d8000 task.ti: ffff9e90369d8000 [93830.784762] RIP: 0010:[&lt;ffffffffc101cc2a&gt;] [&lt;ffffffffc101cc2a&gt;] mdt_object_lock_internal+0x4a/0x360 [mdt] [93830.787609] RSP: 0018:ffff9e90369dba80 EFLAGS: 00010246 [93830.789999] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff9e90369dbae8 [93830.792546] RDX: ffff9e93ef99a0f0 RSI: 0000000000000000 RDI: ffff9e93ef99a000 [93830.795054] RBP: ffff9e90369dbad0 R08: 0000000000000000 R09: 0000000000000000 [93830.797536] R10: ffff9e9400aa3c00 R11: 0000000000000020 R12: ffff9e93ef99a0f0 [93830.799980] R13: ffff9e93ef99a000 R14: ffff9e90369dbae8 R15: 0000000000000000 [93830.802402] FS: 0000000000000000(0000) GS:ffff9e943fc00000(0000) knlGS:0000000000000000 [93830.804901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [93830.807152] CR2: 0000000000000050 CR3: 0000000422b0e000 CR4: 00000000003606f0 [93830.809521] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [93830.811846] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [93830.814138] Call Trace: [93830.816037] [&lt;ffffffffc094c2dc&gt;] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [93830.818297] [&lt;ffffffffc1025f33&gt;] mdt_intent_getxattr+0xb3/0x2c0 [mdt] [93830.820529] [&lt;ffffffffc10229e4&gt;] mdt_intent_policy+0x2d4/0xdd0 [mdt] [93830.822702] [&lt;ffffffffc1025e80&gt;] ? mdt_intent_getattr+0x480/0x480 [mdt] [93830.824902] [&lt;ffffffffc08ffc66&gt;] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [93830.827040] [&lt;ffffffffc05da3d3&gt;] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [93830.829184] [&lt;ffffffffc05dd96e&gt;] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [93830.831278] [&lt;ffffffffc0928587&gt;] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [93830.833470] [&lt;ffffffffc0950520&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [93830.835699] [&lt;ffffffffc09b1082&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] [93830.837786] [&lt;ffffffffc09b72ca&gt;] tgt_request_handle+0x91a/0x15c0 [ptlrpc] [93830.839875] [&lt;ffffffffc05d6fa7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [93830.841956] [&lt;ffffffffc095a88e&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [93830.844066] [&lt;ffffffffb1ccbadb&gt;] ? __wake_up_common+0x5b/0x90 [93830.846014] [&lt;ffffffffc095e384&gt;] ptlrpc_main+0xbb4/0x20f0 [ptlrpc] [93830.847946] [&lt;ffffffffb1cd08c0&gt;] ? finish_task_switch+0x50/0x1c0 [93830.849866] [&lt;ffffffffc095d7d0&gt;] ? ptlrpc_register_service+0xfa0/0xfa0 [ptlrpc] [93830.851842] [&lt;ffffffffb1cc1c71&gt;] kthread+0xd1/0xe0 [93830.853559] [&lt;ffffffffb1cc1ba0&gt;] ? insert_kthread_work+0x40/0x40 [93830.855362] [&lt;ffffffffb2375c1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [93830.857178] [&lt;ffffffffb1cc1ba0&gt;] ? insert_kthread_work+0x40/0x40 [93830.858951] Code: 89 f3 48 83 ec 28 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 f6 05 bd a3 5d ff 01 74 0d f6 05 b8 a3 5d ff 04 0f 85 7e 00 00 00 &lt;48&gt; 8b 43 50 f6 40 1c 02 0f 85 d6 00 00 00 45 0f b6 c9 4d 89 f8 [93830.864078] RIP [&lt;ffffffffc101cc2a&gt;] mdt_object_lock_internal+0x4a/0x360 [mdt] [93830.866025] RSP &lt;ffff9e90369dba80&gt; [93830.867566] CR2: 0000000000000050 </pre> </div></div> <p>In function mdt_object_remote(), there is no check about the pointer o, if the value of o is null, a null pointer dereference bug will happen.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> inline <span class="code-object">int</span> mdt_object_remote(<span class="code-keyword">const</span> struct mdt_object *o){ <span class="code-keyword">return</span> lu_object_remote(&amp;o-&gt;mot_obj); } </pre> </div></div> LU-12615

Lustre mdt_object_remote() bug

Bug Critical Resolved Fixed Hongchao Zhang Alibaba Cloud Tue, 30 Jul 2019 11:19:59 +0000 Thu, 12 Sep 2019 04:14:21 +0000 Wed, 21 Aug 2019 11:29:37 +0000 Lustre 2.13.0 Lustre 2.13.0 Lustre 2.12.3 0 5 <p>Hongchao</p> <p>Could you please investigate</p> <p>Thanks</p> <p>Peter</p> <p>Hi,<br/> What is the client version?<br/> if the mdt_object is NULL, there could be two cases, the first is there is no MDT_BODY field in the request,<br/> the second is no OBD_MD_FLID flag is set on "body-&gt;mbo_valid".</p> <p>btw, is the kernel dump available?<br/> Thanks!</p> <p>Hongchao, this is being induced by specially-crafted RPC requests to test error handling on the server.</p> <p>It looks like this is via:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdt_intent_getxattr() -&gt;mdt_object_lock(info-&gt;mti_object) -&gt;mdt_object_lock_internal() -&gt;mdt_object_remote() -&gt;lu_object_remote(&amp;o-&gt;mot_obj) </pre> </div></div> <p>The <tt>info-&gt;mti_object</tt> field is assigned in <tt>mdt_body_unpack()</tt>, but that is checking <tt>!IS_ERR(obj)</tt> before assigning it. It looks like the problem is caused by the request not containing the <tt>OBD_MD_FLID</tt> flag to indicate that the <tt>mbo_fid1</tt> field is valid, which causes <tt>mdt_body_unpack()</tt> to return without setting <tt>mti_object</tt>, without returning an error. For some RPCs (e.g. <tt>mdt_sync()</tt>) there is an MDT_BODY field, but is not a requirement for <tt>OBD_MD_FLFID</tt> to be set because it can be used for both whole-device sync or single-file sync, so this is correct behavior for <tt>mdt_body_unpack()</tt>. The other place that <tt>mti_object</tt> is assigned is <tt>tsi2mdt_info()</tt> using <tt>tsi_corpus</tt>, but that is itself set in <tt>mdt_body_unpack()</tt> so is the same root cause.</p> <p>I checked all of the <tt>mdt_reint_&#42;</tt> routines and they appear to handle the error from <tt>mdt_object_find()</tt> properly, so you should focus on other places that are using <tt>mti_object</tt> directly. Looking at <tt>mdt_sync()</tt>, the handling is a bit strange because it is checking if <tt>mbo_fid1</tt> is unset for the whole-device sync, then assumes <tt>mti_object</tt> is OK otherwise (which it may not be if <tt>OBD_MD_FLFID</tt> is unset). </p> <p>I think there are two potential ways to handle this:</p> <ul class="alternate" type="square"> <li>add checks for <tt>mti_object == NULL</tt> at the start of high-level RPC handling routines <tt>mdt_intent_getxattr()</tt>, <tt>mdt_getattr()</tt>, <tt>mdt_getxattr()</tt>, <tt>mdt_swap_layouts()</tt>, and <tt>mdt_sync()</tt> (some of which LASSERT(obj != NULL) or equivalent, and should instead just return <tt>-EPROTO</tt> in that case)</li> <li>add checks in low-level routines like <tt>mdt_object_remote()</tt>, <tt>mdt_object_exists()</tt>, <tt>mdt_object_child()</tt>, <tt>mdt_object_fid()</tt> that are accessing <tt>mot_obj</tt>, but those functions are called in a lot of places and if <tt>mti_object</tt> is checked once at the top level then we don't need to check it repeatedly, so I'd rather not do this.</li> </ul> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35764" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35764</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12615" title="Lustre mdt_object_remote() bug" class="issue-link" data-issue-key="LU-12615"><del>LU-12615</del></a> mdt: check mdt_object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9ad3ae7465eaedcc86772c2518e88def4b18b8e8</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35764/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35764/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12615" title="Lustre mdt_object_remote() bug" class="issue-link" data-issue-key="LU-12615"><del>LU-12615</del></a> mdt: check mdt_object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e5e0bdb7a5c2d47ceaa2d1c190806d1be4999129</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35869" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35869</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12615" title="Lustre mdt_object_remote() bug" class="issue-link" data-issue-key="LU-12615"><del>LU-12615</del></a> mdt: check mdt_object<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: f562e6cea802aa408bd94c84427e983eeae55730</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35869/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35869/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12615" title="Lustre mdt_object_remote() bug" class="issue-link" data-issue-key="LU-12615"><del>LU-12615</del></a> mdt: check mdt_object<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 41e2bd9752851a7f3989c6edfe2134742d0ab59f</p> Related LU-12605 Development Rank 1|i00kf3: Rank (Obsolete) 9223372036854775807 Severity