https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5635 Lustre <p>Some hsm operations (lfs hsm_*) were running on a client, and I wanted to check the list of actions:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cat /proc/fs/lustre/mdt/tas01-MDT0000/hsm/actions </pre> </div></div> <p>This led to an immediate crash of the MDT:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;0&gt;LustreError: 3773:0:(llog.c:310:llog_process_thread()) ASSERTION( index &lt;= last_index + 1 ) failed: &lt;0&gt;LustreError: 3773:0:(llog.c:310:llog_process_thread()) LBUG &lt;4&gt;Pid: 3773, comm: cat &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa02f8895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa02f8e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa0413fff&gt;] llog_process_thread+0xd7f/0xdd0 [obdclass] &lt;4&gt; [&lt;ffffffffa0dd7410&gt;] ? hsm_actions_show_cb+0x0/0x3d0 [mdt] &lt;4&gt; [&lt;ffffffffa041585f&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] &lt;4&gt; [&lt;ffffffffa0418612&gt;] llog_cat_process_cb+0x1b2/0x5e0 [obdclass] &lt;4&gt; [&lt;ffffffffa0413c22&gt;] llog_process_thread+0x9a2/0xdd0 [obdclass] &lt;4&gt; [&lt;ffffffff810792c7&gt;] ? current_fs_time+0x27/0x30 &lt;4&gt; [&lt;ffffffffa0418460&gt;] ? llog_cat_process_cb+0x0/0x5e0 [obdclass] &lt;4&gt; [&lt;ffffffffa041585f&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] &lt;4&gt; [&lt;ffffffffa0416b9d&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] &lt;4&gt; [&lt;ffffffffa0dd7410&gt;] ? hsm_actions_show_cb+0x0/0x3d0 [mdt] &lt;4&gt; [&lt;ffffffffa0416d09&gt;] llog_cat_process+0x19/0x20 [obdclass] &lt;4&gt; [&lt;ffffffffa0dd7376&gt;] mdt_hsm_actions_proc_show+0x1a6/0x240 [mdt] &lt;4&gt; [&lt;ffffffffa0dd8b03&gt;] ? mdt_hsm_actions_proc_start+0x63/0x380 [mdt] &lt;4&gt; [&lt;ffffffff811ade02&gt;] seq_read+0xf2/0x400 &lt;4&gt; [&lt;ffffffff8100b9ce&gt;] ? common_interrupt+0xe/0x13 &lt;4&gt; [&lt;ffffffff811f353e&gt;] proc_reg_read+0x7e/0xc0 &lt;4&gt; [&lt;ffffffff81189695&gt;] vfs_read+0xb5/0x1a0 &lt;4&gt; [&lt;ffffffff811897d1&gt;] sys_read+0x51/0x90 &lt;4&gt; [&lt;ffffffff810e1e4e&gt;] ? __audit_syscall_exit+0x25e/0x290 &lt;4&gt; [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b &lt;4&gt; &lt;0&gt;Kernel panic - not syncing: LBUG </pre> </div></div> <p>Crash stack is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; bt PID: 3773 TASK: ffff880130d44ae0 CPU: 3 COMMAND: "cat" #0 [ffff8801144bf900] machine_kexec at ffffffff81038f3b #1 [ffff8801144bf960] crash_kexec at ffffffff810c5d82 #2 [ffff8801144bfa30] panic at ffffffff8152798a #3 [ffff8801144bfab0] lbug_with_loc at ffffffffa02f8eeb [libcfs] #4 [ffff8801144bfad0] llog_process_thread at ffffffffa0413fff [obdclass] #5 [ffff8801144bfb80] llog_process_or_fork at ffffffffa041585f [obdclass] #6 [ffff8801144bfbd0] llog_cat_process_cb at ffffffffa0418612 [obdclass] #7 [ffff8801144bfc30] llog_process_thread at ffffffffa0413c22 [obdclass] #8 [ffff8801144bfce0] llog_process_or_fork at ffffffffa041585f [obdclass] #9 [ffff8801144bfd30] llog_cat_process_or_fork at ffffffffa0416b9d [obdclass] #10 [ffff8801144bfdc0] llog_cat_process at ffffffffa0416d09 [obdclass] #11 [ffff8801144bfde0] mdt_hsm_actions_proc_show at ffffffffa0dd7376 [mdt] #12 [ffff8801144bfe20] seq_read at ffffffff811ade02 #13 [ffff8801144bfea0] proc_reg_read at ffffffff811f353e #14 [ffff8801144bfef0] vfs_read at ffffffff81189695 #15 [ffff8801144bff30] sys_read at ffffffff811897d1 #16 [ffff8801144bff80] system_call_fastpath at ffffffff8100b072 RIP: 00007f6de5e4f730 RSP: 00007fff9aa26d98 RFLAGS: 00000206 RAX: 0000000000000000 RBX: ffffffff8100b072 RCX: 00007f6de5e4f730 RDX: 0000000000008000 RSI: 00000000019c7000 RDI: 0000000000000003 RBP: 00000000019c7000 R8: 00007f6de6103ee8 R9: 0000000000000001 R10: 00007fff9aa26b20 R11: 0000000000000246 R12: ffffffffffff8000 R13: 0000000000000003 R14: 0000000000008000 R15: 0000000000000003 ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b </pre> </div></div> Centos 6.5 with Lustre head of tree (commit f04237) LU-5635

(llog.c:310:llog_process_thread()) ASSERTION( index <= last_index + 1 ) failed:

Bug Minor Resolved Fixed James Nunez Frank Zago patch Wed, 17 Sep 2014 19:04:07 +0000 Wed, 1 Mar 2017 12:00:07 +0000 Thu, 4 Dec 2014 14:14:38 +0000 Lustre 2.6.0 Lustre 2.7.0 0 7 <p>I can reproduce quite easily by running </p> <p> while true ; do cat actions &gt; /dev/null ; done</p> <p>and the crash happens within minutes.</p> <p>I added some traces right before the LASSERT:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>index=64769, last_index=64767, cd=ffff88012144dbe8, last_called_index=64768, cd-&gt;lpcd_first_idx=64768, cd-&gt;lpcd_last_idx=0 LustreError: 9008:0:(llog.c:316:llog_process_thread()) ASSERTION( index &lt;= last_index + 1 ) failed: </pre> </div></div> <p>So the function is entered with index &gt; last_index. Proposed patch: <a href="http://review.whamcloud.com/12161" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12161</a></p> <p>Although the patch does indeed prevent the crash, I'm unsure whether it is the correct place to do the check. May be that should be done earlier in llog_cat_process_cb(), or even in llog_cat_set_first_idx() as that function appears to push index beyond its range.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12161/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12161/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5635" title="(llog.c:310:llog_process_thread()) ASSERTION( index &lt;= last_index + 1 ) failed:" class="issue-link" data-issue-key="LU-5635"><del>LU-5635</del></a> llog: prevent out-of-bound index<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 28a963f266a57308ceb6903da238acf008e628bf</p> <p>Patch landed to master (pre-2.7).</p> <p>The llog_cat_set_first_idx() looks guilty. Consider that there is cancel for last index in bitmap == bitmap_size - 1, i.e. 64767, then cat_idx will be assigned to that value first, then it enters cycle to find first set bit. The problem is that cycle counter 'i' is always inside bitmap because of this:<br/> i = (i + 1) % bitmap_size; but cat_idx is just incremented separately with each iteration, so it will became 64768 and beyond, which is out of range.</p> Related LU-6714 Development Rank 1|hzwwif: Rank (Obsolete) 15772 Severity