https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12469 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 8755.816950] LustreError: 26096:0:(lu_object.c:2399:lu_buf_free()) ASSERTION( buf-&gt;lb_len &gt; 0 ) failed: [ 8755.819785] LustreError: 26096:0:(lu_object.c:2399:lu_buf_free()) LBUG [ 8755.821791] Pid: 26096, comm: mdt00_002 3.10.0-862.14.4.el7_lustre.2.12.54_52_g948202a.x86_64 #1 SMP Thu Jun 13 11:34:55 CST 2019 [ 8755.826429] Call Trace: [ 8755.827263] [&lt;ffffffffc06d77cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 8755.829385] [&lt;ffffffffc06d787c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [ 8755.831108] [&lt;ffffffffc08b9b93&gt;] lu_buf_free+0x213/0x220 [obdclass] [ 8755.832160] [&lt;ffffffffc18227cf&gt;] mdd_iterate_xattrs+0x57f/0x8d0 [mdd] [ 8755.833215] [&lt;ffffffffc184845b&gt;] mdd_declare_migrate_create+0x2bb/0x3b6 [mdd] [ 8755.834365] [&lt;ffffffffc182d810&gt;] mdd_migrate+0xea0/0x1810 [mdd] [ 8755.835344] [&lt;ffffffffc16ec60f&gt;] mdt_reint_migrate+0x1def/0x2400 [mdt] [ 8755.836434] [&lt;ffffffffc16ecca3&gt;] mdt_reint_rec+0x83/0x210 [mdt] [ 8755.837417] [&lt;ffffffffc16c8470&gt;] mdt_reint_internal+0x780/0xb90 [mdt] [ 8755.838474] [&lt;ffffffffc16d3877&gt;] mdt_reint+0x67/0x140 [mdt] [ 8755.839408] [&lt;ffffffffc0f6d60a&gt;] tgt_request_handle+0x91a/0x15c0 [ptlrpc] [ 8755.840575] [&lt;ffffffffc0f117ee&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [ 8755.841810] [&lt;ffffffffc0f152dc&gt;] ptlrpc_main+0xbac/0x1550 [ptlrpc] [ 8755.842865] [&lt;ffffffff8c8bdf21&gt;] kthread+0xd1/0xe0 [ 8755.843706] [&lt;ffffffff8cf255f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [ 8755.844664] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [ 8755.845501] Kernel panic - not syncing: LBUG </pre> </div></div> LU-12469

sanity test_230b crash: LBUG of lu_buf_free in mdd_iterate_xattrs

Bug Major Resolved Fixed Sebastien Buisson Li Xi Tue, 25 Jun 2019 08:13:13 +0000 Sat, 4 Jan 2020 00:02:07 +0000 Sat, 14 Dec 2019 13:42:20 +0000 Lustre 2.14.0 Lustre 2.12.4 0 6 <p>Not sure whether it is related to this LUG, but "xbuf.lb_len" shouldn't be changed, otherwise lu_buf_free() might cause fault memory leak detection.</p> <p>The codes were introduced in "<a href="https://jira.whamcloud.com/browse/LU-8569" title="Sharded DNE directory full of files that don&#39;t exist" class="issue-link" data-issue-key="LU-8569"><del>LU-8569</del></a> linkea: linkEA size limitation"</p> <p>Hit during Maloo testing:<br/> <a href="https://testing.whamcloud.com/test_sets/a9ae171e-fc30-11e9-8e77-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a9ae171e-fc30-11e9-8e77-52540065bddc</a></p> <p>Sebastien, I think the failures on <a href="https://jira.whamcloud.com/browse/LU-12895" title="review-dne-selinux sanity test_185a: lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed" class="issue-link" data-issue-key="LU-12895"><del>LU-12895</del></a> testing are related to those patches, since it hasn't been failing much in other cases (except a bunch of failures for <a href="https://jira.whamcloud.com/browse/LU-12624" title="DNE3: striped directory allocate stripes by QoS" class="issue-link" data-issue-key="LU-12624"><del>LU-12624</del></a> tests that could be attributed to that patch). I don't <em>think</em> it relates to security testing in general, since we are running a lot of that and not seeing this failure. </p> <p>Hmm, unless the <a href="https://jira.whamcloud.com/browse/LU-12895" title="review-dne-selinux sanity test_185a: lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed" class="issue-link" data-issue-key="LU-12895"><del>LU-12895</del></a> crash in test_185a is preventing testing to get to test_230b with the security test config? Is it possible the security config is doing something strange with the buffers and triggering this LBUG() itself?</p> <p>It looks like we are hitting the same assertion for sanity test 230j; <a href="https://testing.whamcloud.com/test_sets/40fb7b16-fff6-11e9-8e77-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/40fb7b16-fff6-11e9-8e77-52540065bddc</a>. </p> <p>The call trace looks a bit different</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10218.643731] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration =================================== 13:23:34 (1572960214) [10218.980145] LustreError: 16021:0:(lu_object.c:2479:lu_buf_free()) ASSERTION( buf-&gt;lb_len &gt; 0 ) failed: [10218.981175] LustreError: 16021:0:(lu_object.c:2479:lu_buf_free()) LBUG [10218.981820] Pid: 16021, comm: mdt00_000 3.10.0-957.27.2.el7_lustre.x86_64 #1 SMP Mon Sep 30 22:09:27 UTC 2019 [10218.982797] Call Trace: [10218.983094] [&lt;ffffffffc06a78ac&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [10218.983793] [&lt;ffffffffc06a795c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [10218.984441] [&lt;ffffffffc08e4f33&gt;] lu_buf_free+0x213/0x220 [obdclass] [10218.985293] [&lt;ffffffffc10cfa20&gt;] mdd_iterate_xattrs+0x5a0/0x8f0 [mdd] [10218.986171] [&lt;ffffffffc10f5251&gt;] mdd_migrate_create+0x280/0x44b [mdd] [10218.986872] [&lt;ffffffffc10da840&gt;] mdd_migrate+0x1290/0x17e0 [mdd] [10218.987518] [&lt;ffffffffc0f67434&gt;] mdt_reint_migrate+0x1c34/0x2230 [mdt] [10218.988285] [&lt;ffffffffc0f67ab3&gt;] mdt_reint_rec+0x83/0x210 [mdt] [10218.988944] [&lt;ffffffffc0f419e0&gt;] mdt_reint_internal+0x7b0/0xba0 [mdt] [10218.989680] [&lt;ffffffffc0f4d6d7&gt;] mdt_reint+0x67/0x140 [mdt] [10218.990319] [&lt;ffffffffc0c2eeea&gt;] tgt_request_handle+0x97a/0x1620 [ptlrpc] [10218.991328] [&lt;ffffffffc0bd5a66&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [10218.992220] [&lt;ffffffffc0bd959c&gt;] ptlrpc_main+0xbac/0x1540 [ptlrpc] [10218.992926] [&lt;ffffffffb12c2e81&gt;] kthread+0xd1/0xe0 [10218.993521] [&lt;ffffffffb1977c37&gt;] ret_from_fork_nospec_end+0x0/0x39 [10218.994190] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [10218.994836] Kernel panic - not syncing: LBUG [10218.995275] CPU: 0 PID: 16021 Comm: mdt00_000 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.27.2.el7_lustre.x86_64 #1 [10218.996449] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [10218.997013] Call Trace: [10218.997283] [&lt;ffffffffb1965147&gt;] dump_stack+0x19/0x1b [10218.997799] [&lt;ffffffffb195e850&gt;] panic+0xe8/0x21f [10218.998278] [&lt;ffffffffc06a79ab&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [10218.998899] [&lt;ffffffffc08e4f33&gt;] lu_buf_free+0x213/0x220 [obdclass] [10218.999525] [&lt;ffffffffc10cfa20&gt;] mdd_iterate_xattrs+0x5a0/0x8f0 [mdd] [10219.000168] [&lt;ffffffffc10cb700&gt;] ? mdo_ref_del+0x180/0x180 [mdd] [10219.000790] [&lt;ffffffffc10f5251&gt;] mdd_migrate_create+0x280/0x44b [mdd] [10219.001442] [&lt;ffffffffc10da840&gt;] mdd_migrate+0x1290/0x17e0 [mdd] [10219.002058] [&lt;ffffffffc0f67434&gt;] mdt_reint_migrate+0x1c34/0x2230 [mdt] [10219.002712] [&lt;ffffffffc0f67ab3&gt;] mdt_reint_rec+0x83/0x210 [mdt] [10219.003329] [&lt;ffffffffc0f419e0&gt;] mdt_reint_internal+0x7b0/0xba0 [mdt] [10219.003988] [&lt;ffffffffc0f4af67&gt;] ? mdt_thread_info_init+0xa7/0x1e0 [mdt] [10219.004663] [&lt;ffffffffc0f4d6d7&gt;] mdt_reint+0x67/0x140 [mdt] [10219.005257] [&lt;ffffffffc0c2eeea&gt;] tgt_request_handle+0x97a/0x1620 [ptlrpc] [10219.005989] [&lt;ffffffffc07f0f3e&gt;] ? libcfs_nid2str_r+0xfe/0x130 [lnet] [10219.006661] [&lt;ffffffffc0bd5a66&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [10219.007423] [&lt;ffffffffb12cfeb4&gt;] ? __wake_up+0x44/0x50 [10219.007968] [&lt;ffffffffc0bd959c&gt;] ptlrpc_main+0xbac/0x1540 [ptlrpc] [10219.008613] [&lt;ffffffffc0bd89f0&gt;] ? ptlrpc_register_service+0xf90/0xf90 [ptlrpc] [10219.009346] [&lt;ffffffffb12c2e81&gt;] kthread+0xd1/0xe0 [10219.009825] [&lt;ffffffffb12c2db0&gt;] ? insert_kthread_work+0x40/0x40 [10219.010429] [&lt;ffffffffb1977c37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [10219.011077] [&lt;ffffffffb12c2db0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36675" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36675</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12469" title="sanity test_230b crash: LBUG of lu_buf_free in mdd_iterate_xattrs" class="issue-link" data-issue-key="LU-12469"><del>LU-12469</del></a> tests: stop running sanity test 230j<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d8e6320e599012bcdebae46124006a8ee6ed3a76</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36684" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36684</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12469" title="sanity test_230b crash: LBUG of lu_buf_free in mdd_iterate_xattrs" class="issue-link" data-issue-key="LU-12469"><del>LU-12469</del></a> mdd: handle migrate case with SELinux<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2192fcce2f7469c4d78b398ecde9ffa1faf1995f</p> <p>The stack trace reported for crash in sanity test_230j is quite similar to the one for crash in test_230b, and is much likely happening because of SELinux.</p> <p>As can be seen in test results for patch <a href="https://review.whamcloud.com/36684" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36684</a>, all sanity tests in the 230 family are passing now. So I do not think patch <a href="https://review.whamcloud.com/36675" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36675</a> to add sanity 230j to the exception list is necessary.</p> <p>+1 during sanity/test_230e with latest master @<a href="https://testing.whamcloud.com/test_sets/f7c46c1e-055e-11ea-b934-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f7c46c1e-055e-11ea-b934-52540065bddc</a></p> <p>+1 during sanity/test_230l with latest master @<a href="https://testing.whamcloud.com/test_sets/af8d3b8c-0569-11ea-b934-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/af8d3b8c-0569-11ea-b934-52540065bddc</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36684/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36684/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12469" title="sanity test_230b crash: LBUG of lu_buf_free in mdd_iterate_xattrs" class="issue-link" data-issue-key="LU-12469"><del>LU-12469</del></a> mdd: handle migrate case with SELinux<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8a60fa2e2fcd28c2772d90e76d36430d30b01905</p> <p>Landed for 2.14</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37031" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37031</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12469" title="sanity test_230b crash: LBUG of lu_buf_free in mdd_iterate_xattrs" class="issue-link" data-issue-key="LU-12469"><del>LU-12469</del></a> mdd: handle migrate case with SELinux<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a977e99219688feb4e1e60ea687f02ef2008fab3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37031/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37031/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12469" title="sanity test_230b crash: LBUG of lu_buf_free in mdd_iterate_xattrs" class="issue-link" data-issue-key="LU-12469"><del>LU-12469</del></a> mdd: handle migrate case with SELinux<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: f9340d5eb4c8e74909a1aec206844dc4d0f30ec1</p> Related LU-11956 Development Rank 1|i00iof: Rank (Obsolete) 9223372036854775807