https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2843 Lustre <p>Steps leading up to:</p> <ol> <li>create new filesystem</li> <li>register changelog user</li> <li>run createmany and unlinkmany in infinite loop until catalog full (<a href="https://jira.whamcloud.com/browse/LU-1586" title="no free catalog slots for log" class="issue-link" data-issue-key="LU-1586"><del>LU-1586</del></a>)</li> <li>lctl --device lustre-MDT0000 changelog_deregister cl1</li> </ol> <p>Build contains patches for <a href="https://jira.whamcloud.com/browse/LU-2129" title="ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index )" class="issue-link" data-issue-key="LU-2129"><del>LU-2129</del></a> and <a href="https://jira.whamcloud.com/browse/LU-2102" title="changelog_user_init_cb()) ASSERTION( rec-&gt;cur_hdr.lrh_type == CHANGELOG_USER_REC ) " class="issue-link" data-issue-key="LU-2102"><del>LU-2102</del></a>.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 30248:0:(llog_osd.c:596:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed: LustreError: 30248:0:(llog_osd.c:596:llog_osd_next_block()) LBUG Pid: 30248, comm: lctl </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 30248 TASK: ffff88004bd7e080 CPU: 3 COMMAND: "lctl" #0 [ffff88005029b590] machine_kexec at ffffffff8103283b #1 [ffff88005029b5f0] crash_kexec at ffffffff810ba492 #2 [ffff88005029b6c0] panic at ffffffff814fdcf2 #3 [ffff88005029b740] lbug_with_loc at ffffffffa0fc7fbb [libcfs] #4 [ffff88005029b760] llog_osd_next_block at ffffffffa06ba3ed [obdclass] #5 [ffff88005029b800] llog_process_thread at ffffffffa068a7f6 [obdclass] #6 [ffff88005029b8b0] llog_process_or_fork at ffffffffa068cb0d [obdclass] #7 [ffff88005029b900] llog_cat_process_cb at ffffffffa068f282 [obdclass] #8 [ffff88005029b950] llog_process_thread at ffffffffa068adfb [obdclass] #9 [ffff88005029ba00] llog_process_or_fork at ffffffffa068cb0d [obdclass] #10 [ffff88005029ba50] llog_cat_process_or_fork at ffffffffa068d5b9 [obdclass] #11 [ffff88005029bae0] llog_cat_process at ffffffffa068d7c9 [obdclass] #12 [ffff88005029bb00] llog_changelog_cancel at ffffffffa0c730ff [mdd] #13 [ffff88005029bb40] llog_cancel at ffffffffa0691538 [obdclass] #14 [ffff88005029bb80] mdd_changelog_llog_cancel at ffffffffa0c77dae [mdd] #15 [ffff88005029bbc0] mdd_changelog_user_purge at ffffffffa0c78220 [mdd] #16 [ffff88005029bc10] mdd_iocontrol at ffffffffa0c78776 [mdd] #17 [ffff88005029bc70] mdt_ioc_child at ffffffffa0d50129 [mdt] #18 [ffff88005029bcf0] mdt_iocontrol at ffffffffa0d54608 [mdt] #19 [ffff88005029bd90] class_handle_ioctl at ffffffffa069ba4f [obdclass] #20 [ffff88005029be40] obd_class_ioctl at ffffffffa06852ab [obdclass] #21 [ffff88005029be60] vfs_ioctl at ffffffff8118ddb2 #22 [ffff88005029bea0] do_vfs_ioctl at ffffffff8118df54 #23 [ffff88005029bf30] sys_ioctl at ffffffff8118e4d1 #24 [ffff88005029bf80] system_call_fastpath at ffffffff8100b0f2 RIP: 0000003ae74dde07 RSP: 00007fff6c48fc00 RFLAGS: 00010246 RAX: 0000000000000010 RBX: ffffffff8100b0f2 RCX: 0000000000000000 RDX: 00007fff6c48fc40 RSI: 00000000424066b2 RDI: 0000000000000003 RBP: 0000000000000001 R8: 0000000000000000 R9: 0000000000000240 R10: 0000000000000000 R11: 0000000000000246 R12: 00000000424066b2 R13: 00007fff6c48fc40 R14: 0000000000678380 R15: 0000000000000003 ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b </pre> </div></div> 2.6.32-279.9.1.1chaos.ch5.1.x86_64<br/> KVM x86_64 virtual machine LU-2843

ASSERTION( last_rec->lrh_index == tail->lrt_index )

Bug Blocker Resolved Fixed Hongchao Zhang Ned Bass MB Wed, 20 Feb 2013 17:42:27 +0000 Wed, 3 Apr 2013 16:50:28 +0000 Wed, 3 Apr 2013 16:50:28 +0000 Lustre 2.4.0 Lustre 2.4.0 0 4 <p>Hongchao</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>It makes sense to add debug into llog_osd_write_rec() to see how log records were written upon llog ends. There are several separated writes and maybe we have situation when part of data is written and other is not. Assertion in llog_osd_next_block() is just result of failed write most probably</p> <p>status update: <br/> the issue has been reproduced locally and the patch is under creation &amp; test.</p> <p>Hongchao, could you decribe the reason of issue? We saw similar one in Orion but it gone at some point. I wonder what can be root cause now.</p> <p>the patch is <a href="http://review.whamcloud.com/#change,5604" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5604</a></p> <p>this issue is caused by the ways of writing LLOG record, one record is written into LLOG by 3 continuous write operations at most,<br/> if the second or third write failed, then the data written previously will damage the LLOG for it did not know the exact size of<br/> the LLOG file and it will treat all data it can read as LLOG records.</p> <p>yes, that is technical debt, those 3-step blob write can be partial.</p> <p>the single write can also cause this problem,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> osd_ldiskfs_write_record(struct inode *inode, void *buf, <span class="code-object">int</span> bufsize, <span class="code-object">int</span> write_NUL, loff_t *offs, handle_t *handle) { ... <span class="code-keyword">while</span> (bufsize &gt; 0) { <span class="code-keyword">if</span> (bh != NULL) brelse(bh); block = offset &gt;&gt; inode-&gt;i_blkbits; boffs = offset &amp; (blocksize - 1); size = min(blocksize - boffs, bufsize); bh = ldiskfs_bread(handle, inode, block, 1, &amp;err); &lt;---------- could fail <span class="code-keyword">for</span> -ENOSPC <span class="code-keyword">if</span> (!bh) { CERROR(<span class="code-quote">"%s: error reading offset %llu (block %lu): "</span> <span class="code-quote">"rc = %d\n"</span>, inode-&gt;i_sb-&gt;s_id, offset, block, err); <span class="code-keyword">break</span>; } ... } ... } </pre> </div></div> <p>currently, osd_punch will cause LBUG "LASSERT(op &lt; OSD_OT_MAX)" in "osd_trans_exec_op"</p> <p>it can fail, yes, but it doesn't change i_size so no need to return it back. As for OSD_OT_MAX - it need just be increased if such situation is possible and valid</p> <p>no, it change the size even on the failed case(-ENOSPC)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-comment">/* correct in-core and on-disk sizes */</span> <span class="code-keyword">if</span> (new_size &gt; i_size_read(inode)) { spin_lock(&amp;inode-&gt;i_lock); <span class="code-keyword">if</span> (new_size &gt; i_size_read(inode)) i_size_write(inode, new_size); <span class="code-keyword">if</span> (i_size_read(inode) &gt; LDISKFS_I(inode)-&gt;i_disksize) { LDISKFS_I(inode)-&gt;i_disksize = i_size_read(inode); dirty_inode = 1; } spin_unlock(&amp;inode-&gt;i_lock); <span class="code-keyword">if</span> (dirty_inode) inode-&gt;i_sb-&gt;s_op-&gt;dirty_inode(inode); } </pre> </div></div> <p>does we fix here by not change the size for error case (could it affect other cases which need this behavior) or fix it in the LLOG layer?</p> <p>check new_size value, it is not changed in case of failed ldiskfs_bread() so we will not enter under that 'if' clause. That would be just OSD bug if size is changed upon failed write.</p> <p>Landed for 2.4</p> Development Rank 1|hzvjjj: Rank (Obsolete) 6882 Severity