https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9848 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 43470:0:(llog_osd.c:165:llog_osd_pad()) ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 ) failed: LustreError: 43470:0:(llog_osd.c:165:llog_osd_pad()) LBUG Pid: 43470, comm: mdt00_081 Kernel panic - not syncing: LBUG CPU: 2 PID: 43470 Comm: mdt00_081 Tainted: P OE ------------ 3.10.0-514.26.2.1chaos.ch6_1.x86_64 #1 Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016 ffffffffa0c50e0f 000000005a5d14f4 ffff887dd2457700 ffffffff8169d4bc ffff887dd2457780 ffffffff816966ff ffffffff00000008 ffff887dd2457790 ffff887dd2457730 000000005a5d14f4 ffffffffa0d861e7 0000000000000246 Call Trace: [&lt;ffffffff8169d4bc&gt;] dump_stack+0x19/0x1b [&lt;ffffffff816966ff&gt;] panic+0xe3/0x1f2 [&lt;ffffffff810a6ac0&gt;] ? call_usermodehelper_freeinfo+0x20/0x30 [&lt;ffffffffa0c34deb&gt;] lbug_with_loc+0xab/0xc0 [libcfs] [&lt;ffffffffa0d1727a&gt;] llog_osd_pad+0x3ca/0x440 [obdclass] [&lt;ffffffffa0d19967&gt;] llog_osd_write_rec+0xe87/0x14d0 [obdclass] [&lt;ffffffffa0d0b8da&gt;] llog_write_rec+0xaa/0x280 [obdclass] [&lt;ffffffffa0d100c0&gt;] llog_cat_add_rec+0x210/0x8e0 [obdclass] [&lt;ffffffffa0d08a3a&gt;] llog_add+0x7a/0x1a0 [obdclass] [&lt;ffffffffa1029f7c&gt;] ? sub_updates_write+0x7f6/0xef8 [ptlrpc] [&lt;ffffffffa102a373&gt;] sub_updates_write+0xbed/0xef8 [ptlrpc] [&lt;ffffffffa101899f&gt;] top_trans_stop+0x62f/0x970 [ptlrpc] [&lt;ffffffffa134c399&gt;] lod_trans_stop+0x259/0x340 [lod] [&lt;ffffffffa13c0c32&gt;] ? mdd_links_rename+0x312/0x5d0 [mdd] [&lt;ffffffffa13daafd&gt;] mdd_trans_stop+0x1d/0x25 [mdd] [&lt;ffffffffa13c5c18&gt;] mdd_link+0x2e8/0x930 [mdd] [&lt;ffffffffa0fa42d2&gt;] ? lustre_msg_get_versions+0x22/0xf0 [ptlrpc] [&lt;ffffffffa1296b6e&gt;] mdt_reint_link+0xade/0xc30 [mdt] [&lt;ffffffff8132f4d2&gt;] ? strlcpy+0x42/0x60 [&lt;ffffffffa1298ef0&gt;] mdt_reint_rec+0x80/0x210 [mdt] [&lt;ffffffffa127bdf1&gt;] mdt_reint_internal+0x5e1/0x990 [mdt] [&lt;ffffffffa1285a07&gt;] mdt_reint+0x67/0x140 [mdt] [&lt;ffffffffa1004425&gt;] tgt_request_handle+0x915/0x1320 [ptlrpc] [&lt;ffffffffa0fb0e7b&gt;] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] [&lt;ffffffffa0c41748&gt;] ? lc_watchdog_touch+0x68/0x180 [libcfs] [&lt;ffffffffa0faea4b&gt;] ? ptlrpc_wait_event+0xab/0x350 [ptlrpc] [&lt;ffffffff810c8aa2&gt;] ? default_wake_function+0x12/0x20 [&lt;ffffffff810bdb18&gt;] ? __wake_up_common+0x58/0x90 [&lt;ffffffffa0fb4f20&gt;] ptlrpc_main+0xa90/0x1db0 [ptlrpc] [&lt;ffffffff8102a569&gt;] ? __switch_to+0xd9/0x4e0 [&lt;ffffffffa0fb4490&gt;] ? ptlrpc_register_service+0xe40/0xe40 [ptlrpc] [&lt;ffffffff810b3a0f&gt;] kthread+0xcf/0xe0 </pre> </div></div> <p>Appears to be regression of <a href="https://jira.whamcloud.com/browse/LU-8422" title="llog_osd.c:165:llog_osd_pad()) ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 )" class="issue-link" data-issue-key="LU-8422"><del>LU-8422</del></a></p> Linux version 3.10.0-514.26.2.1chaos.ch6_1.x86_64<br/> Lustre 2.8.0_9.chaos<br/> ZFS v0.7.0-8_g0f976d6 LU-9848

LBUG: ASSERTION( len >= (24) && (len & 0x7) == 0 ) failed

Bug Major Resolved Fixed Lai Siyao Olaf Faaland llnl Tue, 8 Aug 2017 23:06:15 +0000 Mon, 18 Sep 2017 21:30:41 +0000 Mon, 28 Aug 2017 14:53:18 +0000 Lustre 2.10.1 Lustre 2.11.0 0 7 <p>Intel has access to the patch stack we are running. See the lustre-release-fe-llnl project in gerrit.</p> <p>I'm able to reproduce this with a sizable simul run that only runs the symlink tests. Something similar to the following hits the issue within minutes on our testbed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>srun -N 72 -n $((72*36)) ./simul -d /p/lquake/dinatale/SIMUL -V 1 -n 20 -i "16,36,38,39,12,18,19,32" </pre> </div></div> <p>Lai</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Lai,</p> <p>We found that the directory the test, /p/lquake/dinatale/SIMUL/, is sharded (DNE2). We only run DNE1 in production and that may explain why we suddenly started encountering the error. We have no plans at this time to use DNE2 in production and so this may not need attention at all.</p> <p>We will re-test with DNE1 directories and update this ticket.</p> <p>I ran the test on DNE1 directories and we don't hit the assertion. We can leave this ticket open since I believe the assert is still a problem, but we won't be hitting this in production.</p> <p>What would be interesting to know is whether this issue still hits on the latest master (or at least 2.10.x)</p> <p>Peter, are you able to run those tests on your test hardware?</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28554" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28554</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9848" title="LBUG: ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 ) failed" class="issue-link" data-issue-key="LU-9848"><del>LU-9848</del></a> llog: check padding size for update reclen<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 254ac377c0a0dff333cd12bdc89a128a67648968</p> <p>This is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-8422" title="llog_osd.c:165:llog_osd_pad()) ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 )" class="issue-link" data-issue-key="LU-8422"><del>LU-8422</del></a>, but I think the fix <a href="https://review.whamcloud.com/#/c/21509/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/21509/</a> for <a href="https://jira.whamcloud.com/browse/LU-8422" title="llog_osd.c:165:llog_osd_pad()) ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 )" class="issue-link" data-issue-key="LU-8422"><del>LU-8422</del></a> is not quite right, so I committed a new fix here.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28554/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28554/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9848" title="LBUG: ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 ) failed" class="issue-link" data-issue-key="LU-9848"><del>LU-9848</del></a> llog: check padding size for update reclen<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6705d6cc40a83b0e94668d651c444c855482bd01</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28762" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28762</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9848" title="LBUG: ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 ) failed" class="issue-link" data-issue-key="LU-9848"><del>LU-9848</del></a> llog: check padding size for update reclen<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 8d1a5d38b16ecde701efc6b0807291d9f76268e7</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/28762/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28762/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9848" title="LBUG: ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 ) failed" class="issue-link" data-issue-key="LU-9848"><del>LU-9848</del></a> llog: check padding size for update reclen<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 18582ce8e6fbfd864bafbb7f92246f7a21f1681a</p> Related LU-8422 Development Rank 1|hzzi4n: Rank (Obsolete) 9223372036854775807 Severity