https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12707 Lustre <p>mds-survey test_1 started crashing with a NULL pointer dereference in echo_object_free() on 16 August 2019. </p> <p>Looking at the kernel-crash log, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[37813.223580] Lustre: DEBUG MARKER: == mds-survey test 1: Metadata survey with zero-stripe =============================================== 15:50:41 (1565970641) [37813.430124] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [37814.598393] Lustre: Echo OBD driver; http://www.lustre.org/ [37820.151690] LustreError: 14608:0:(echo_client.c:1821:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [37820.152651] LustreError: 14608:0:(echo_client.c:2055:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [37820.674040] LustreError: 14674:0:(echo_client.c:1821:echo_md_lookup()) lookup MDT0000-tests2: rc = -2 [37820.675015] LustreError: 14674:0:(echo_client.c:1821:echo_md_lookup()) Skipped 1 previous similar message [37820.675915] LustreError: 14674:0:(echo_client.c:2055:echo_md_destroy_internal()) Can't find child MDT0000-tests2: rc = -2 [37820.676936] LustreError: 14674:0:(echo_client.c:2055:echo_md_destroy_internal()) Skipped 1 previous similar message [37822.418041] LustreError: 14873:0:(echo_client.c:1821:echo_md_lookup()) lookup MDT0002-tests: rc = -2 [37822.419018] LustreError: 14873:0:(echo_client.c:1821:echo_md_lookup()) Skipped 1 previous similar message [37822.419931] LustreError: 14873:0:(echo_client.c:2055:echo_md_destroy_internal()) Can't find child MDT0002-tests: rc = -2 [37822.421048] LustreError: 14873:0:(echo_client.c:2055:echo_md_destroy_internal()) Skipped 1 previous similar message [39010.965720] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040 [39010.966793] IP: [&lt;ffffffffc102ad18&gt;] echo_object_free+0x28/0x490 [obdecho] [39010.967550] PGD 8000000016975067 PUD 16976067 PMD 0 [39010.968086] Oops: 0000 [#1] SMP [39010.968876] Modules linked in: obdecho(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core sunrpc dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon parport_pc parport i2c_piix4 ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk ata_piix libata crct10dif_pclmul crct10dif_common [39010.976913] crc32c_intel 8139too serio_raw virtio_pci 8139cp virtio_ring virtio mii floppy [39010.977773] CPU: 0 PID: 16251 Comm: lctl Kdump: loaded Tainted: G OE ------------ 3.10.0-957.21.3.el7_lustre.x86_64 #1 [39010.978869] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [39010.979407] task: ffff88d947864100 ti: ffff88d936efc000 task.ti: ffff88d936efc000 [39010.980101] RIP: 0010:[&lt;ffffffffc102ad18&gt;] [&lt;ffffffffc102ad18&gt;] echo_object_free+0x28/0x490 [obdecho] [39010.980993] RSP: 0018:ffff88d936effb18 EFLAGS: 00010246 [39010.981484] RAX: 0000000000000000 RBX: ffff88d93aaa4cc0 RCX: dead000000000200 [39010.982152] RDX: ffff88d936effb40 RSI: ffff88d93aaa4cc0 RDI: ffff88d917df92d8 [39010.982815] RBP: ffff88d936effb30 R08: ffff88d93aaa4cd8 R09: 0000000000000010 [39010.983481] R10: 0000000000000223 R11: 000000000007ffff R12: ffff88d93aaa4cd8 [39010.984149] R13: ffff88d917df92d8 R14: ffff88d936effb40 R15: ffffa34d86641028 [39010.984810] FS: 00007fd52a7ec740(0000) GS:ffff88d97fc00000(0000) knlGS:0000000000000000 [39010.985566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39010.986113] CR2: 0000000000000040 CR3: 0000000044ab4000 CR4: 00000000000606f0 [39010.986779] Call Trace: [39010.987156] [&lt;ffffffffc0891592&gt;] lu_object_free.isra.32+0xf2/0x170 [obdclass] [39010.987857] [&lt;ffffffffc0895756&gt;] lu_object_find_at+0x5f6/0xa60 [obdclass] [39010.988527] [&lt;ffffffffa9db91de&gt;] ? filemap_fault+0x17e/0x490 [39010.989080] [&lt;ffffffffc1034012&gt;] echo_md_handler.isra.43+0x362/0x2c60 [obdecho] [39010.989769] [&lt;ffffffffc10377a8&gt;] echo_client_iocontrol+0xe98/0x1860 [obdecho] [39010.990894] [&lt;ffffffffc085cbd0&gt;] ? obd_ioctl_getdata+0x200/0x11b0 [obdclass] [39010.991583] [&lt;ffffffffc085f4aa&gt;] class_handle_ioctl+0x192a/0x1e30 [obdclass] [39010.992268] [&lt;ffffffffa9dec27d&gt;] ? handle_mm_fault+0x39d/0x9b0 [39010.992836] [&lt;ffffffffa9ef8fde&gt;] ? security_capable+0x1e/0x20 [39010.993405] [&lt;ffffffffc085fa25&gt;] obd_class_ioctl+0x75/0x170 [obdclass] [39010.994038] [&lt;ffffffffa9e569d0&gt;] do_vfs_ioctl+0x3a0/0x5a0 [39010.994572] [&lt;ffffffffa9e56c71&gt;] SyS_ioctl+0xa1/0xc0 [39010.995075] [&lt;ffffffffaa375d15&gt;] ? system_call_after_swapgs+0xa2/0x146 [39010.995695] [&lt;ffffffffaa375ddb&gt;] system_call_fastpath+0x22/0x27 [39010.996265] [&lt;ffffffffaa375d21&gt;] ? system_call_after_swapgs+0xae/0x146 [39010.996875] Code: 00 00 00 66 66 66 66 90 55 48 85 f6 48 89 e5 41 55 41 54 53 48 89 f3 0f 85 1d 04 00 00 f6 05 c7 22 65 ff 01 48 8b 83 98 00 00 00 &lt;4c&gt; 8b 60 40 0f 85 8e 01 00 00 8b 83 b8 00 00 00 85 c0 0f 85 c5 [39011.000093] RIP [&lt;ffffffffc102ad18&gt;] echo_object_free+0x28/0x490 [obdecho] [39011.000758] RSP &lt;ffff88d936effb18&gt; [39011.001102] CR2: 0000000000000040 </pre> </div></div> <p>This test is crashing for ldiskfs, ZFS, DNE and non-DNE environments.</p> <p>Here are links to logs for several crashes:<br/> <a href="https://testing.whamcloud.com/test_sets/b53bc6f2-c07c-11e9-9fc9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b53bc6f2-c07c-11e9-9fc9-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/7512ef7a-bfe0-11e9-90ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7512ef7a-bfe0-11e9-90ad-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/b35271e6-c0e1-11e9-a2b6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b35271e6-c0e1-11e9-a2b6-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/25c95ba8-c15b-11e9-a2b6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/25c95ba8-c15b-11e9-a2b6-52540065bddc</a></p> LU-12707

mds-survey test 1 crashes with a NULL pointer dereference in echo_object_free()

Bug Minor Resolved Fixed Alexey Lyashkov James Nunez Tue, 27 Aug 2019 20:27:41 +0000 Mon, 10 Feb 2020 16:44:24 +0000 Fri, 20 Sep 2019 14:44:07 +0000 Lustre 2.13.0 Lustre 2.12.3 Lustre 2.13.0 Lustre 2.12.4 0 6 <p>The most obvious candidate is patch <a href="https://review.whamcloud.com/35700" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35700</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12578" title="obdfilter-survey w/echo-client does not gain from direct-io optimizations" class="issue-link" data-issue-key="LU-12578"><del>LU-12578</del></a> obdecho: reuse an cl env cache for obdecho survey</tt>", which landed on 2019-08-15.</p> <p>Alex, Alexey, any thoughts on this crash? It appears related to the <a href="https://jira.whamcloud.com/browse/LU-12578" title="obdfilter-survey w/echo-client does not gain from direct-io optimizations" class="issue-link" data-issue-key="LU-12578"><del>LU-12578</del></a> patch at first glance.</p> <p>Andreas, I think it fail don't related to the <a href="https://jira.whamcloud.com/browse/LU-12578" title="obdfilter-survey w/echo-client does not gain from direct-io optimizations" class="issue-link" data-issue-key="LU-12578"><del>LU-12578</del></a> patch.</p> <p>looking to the obdecho code I found strange/bug in echo_resolve_path </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (gdb) l *(echo_md_handler+0x71c) 0xaccc is in echo_md_handler (lustre/obdecho/echo_client.c:2202). 2197 rc = -EINVAL; 2198 <span class="code-keyword">break</span>; 2199 } 2200 2201 child = echo_md_lookup(env, ed, lu2md(ld_parent), lname); 2202 lu_object_put(env, parent); 2203 <span class="code-keyword">if</span> (IS_ERR(child)) { 2204 rc = (<span class="code-object">int</span>)PTR_ERR(child); 2205 CERROR(<span class="code-quote">"lookup %s under parent "</span>DFID<span class="code-quote">": rc = %d\n"</span>, 2206 lname-&gt;ln_name, PFID(lu_object_fid(ld_parent)), (gdb) </pre> </div></div> <p>it looks CERROR function tries to make access to the pointer after last put which called before 'IF'.</p> <p>I have no idea who resolve path can't found a work dir as it run short after lu_env_refill() so it have guarantee lu_env filled correctly.</p> <p>can someone change a lu_object_put place and check how it helps for Mallo ?</p> <p>I able to replicate it bug. Will provide some info short later, based on my logs - lu_object_find_at start to free an object where it don't have 'init' passed and it caused a null pointer deref. <br/> good case</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000001:0.0:1566998598.740494:0:20147:0:(lu_object.c:735:lu_object_find_at()) Process entered 00008000:00000001:0.0:1566998598.740495:0:20147:0:(echo_client.c:580:echo_object_alloc()) Process entered 00008000:00000010:0.0:1566998598.740495:0:20147:0:(echo_client.c:583:echo_object_alloc()) slab-alloced 'eco': 264 at ffff88003c438840. 00000020:00000001:0.0:1566998598.740496:0:20147:0:(cl_object.c:72:cl_object_header_init()) Process entered 00000020:00000001:0.0:1566998598.740498:0:20147:0:(cl_object.c:79:cl_object_header_init()) Process leaving (rc=0 : 0 : 0) 00008000:00000001:0.0:1566998598.740499:0:20147:0:(echo_client.c:597:echo_object_alloc()) Process leaving (rc=18446612133325277248 : -131940384274368 : ffff88003c438840) 00008000:00000001:0.0:1566998598.740500:0:20147:0:(echo_client.c:487:echo_object_init()) Process entered 00000004:00000010:0.0:1566998598.740501:0:20147:0:(mdd_object.c:282:mdd_object_alloc()) slab-alloced 'mdd_obj': 96 at ffff88003b6e5e40. 00008000:00000001:0.0:1566998598.740502:0:20147:0:(echo_client.c:524:echo_object_init()) Process leaving (rc=0 : 0 : 0) 00000004:00000001:0.0:1566998598.740503:0:20147:0:(mdd_object.c:304:mdd_object_init()) Process entered </pre> </div></div> <p>bad case</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000001:0.0:1566998604.027293:0:20164:0:(lu_object.c:735:lu_object_find_at()) Process entered 00008000:00000001:0.0:1566998604.027294:0:20164:0:(echo_client.c:580:echo_object_alloc()) Process entered 00008000:00000010:0.0:1566998604.027295:0:20164:0:(echo_client.c:583:echo_object_alloc()) slab-alloced 'eco': 264 at ffff88003b61a108. 00000020:00000001:0.0:1566998604.027296:0:20164:0:(cl_object.c:72:cl_object_header_init()) Process entered 00000020:00000001:0.0:1566998604.027298:0:20164:0:(cl_object.c:79:cl_object_header_init()) Process leaving (rc=0 : 0 : 0) 00008000:00000001:0.0:1566998604.027299:0:20164:0:(echo_client.c:597:echo_object_alloc()) Process leaving (rc=18446612133310472456 : -131940399079160 : ffff88003b61a108) &gt;&gt;&gt;&gt;&gt; next line is panic in lu_object_free </pre> </div></div> <p>Second question - why "echo_md_lookup()) lookup MDT0000-tests: rc = -2" hit, don't invested now.</p> <p>provided patch allow md-survey to be finished fine.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/obdecho/echo_client.c b/lustre/obdecho/echo_client.c index 11191fb055..1970d17bc5 100644 --- a/lustre/obdecho/echo_client.c +++ b/lustre/obdecho/echo_client.c @@ -527,14 +527,18 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> echo_object_init(<span class="code-keyword">const</span> struct lu_env *env, struct lu_object *obj, <span class="code-keyword">static</span> void echo_object_free(<span class="code-keyword">const</span> struct lu_env *env, struct lu_object *obj) { struct echo_object *eco = cl2echo_obj(lu2cl(obj)); - struct echo_client_obd *ec = eco-&gt;eo_dev-&gt;ed_ec; + struct echo_client_obd *ec; ENTRY; - LASSERT(atomic_read(&amp;eco-&gt;eo_npages) == 0); + <span class="code-keyword">if</span> (eco-&gt;eo_dev != NULL) { + ec = eco-&gt;eo_dev-&gt;ed_ec; - spin_lock(&amp;ec-&gt;ec_lock); - list_del_init(&amp;eco-&gt;eo_obj_chain); - spin_unlock(&amp;ec-&gt;ec_lock); + LASSERT(atomic_read(&amp;eco-&gt;eo_npages) == 0); + + spin_lock(&amp;ec-&gt;ec_lock); + list_del_init(&amp;eco-&gt;eo_obj_chain); + spin_unlock(&amp;ec-&gt;ec_lock); + } lu_object_fini(obj); lu_object_header_fini(obj-&gt;lo_header); </pre> </div></div> <p>Alexey Lyashkov (c17817@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35950" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35950</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12707" title="mds-survey test 1 crashes with a NULL pointer dereference in echo_object_free()" class="issue-link" data-issue-key="LU-12707"><del>LU-12707</del></a> obdecho: avoid panic with partially object init<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dedb59b36b7e243dd1a27e112838de1f67412a10</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35950/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35950/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12707" title="mds-survey test 1 crashes with a NULL pointer dereference in echo_object_free()" class="issue-link" data-issue-key="LU-12707"><del>LU-12707</del></a> obdecho: avoid panic with partially object init<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1a9ca8417c60f04a9aa719b7254372e2d18a6b0a</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36317" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36317</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12707" title="mds-survey test 1 crashes with a NULL pointer dereference in echo_object_free()" class="issue-link" data-issue-key="LU-12707"><del>LU-12707</del></a> obdecho: avoid panic with partially object init<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a4284f3cd19e8b4f5f52df923869513209e32eb6</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36317/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36317/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12707" title="mds-survey test 1 crashes with a NULL pointer dereference in echo_object_free()" class="issue-link" data-issue-key="LU-12707"><del>LU-12707</del></a> obdecho: avoid panic with partially object init<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 362e9365699abf08d17044c3c9dd60acfb332a9a</p> Related LU-12034 LU-12578 Development Rank 1|i00lyf: Rank (Obsolete) 9223372036854775807 Severity