https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4382 Lustre <p>Running lustre <a href="https://github.com/chaos/lustre/commits/2.4.0-19chaos" class="external-link" target="_blank" rel="nofollow noopener">2.4.0-19chaos</a> we recently had several OSS nodes crass on the following BUG:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kernel BUG at fs/jbd2/transaction.c:1033</pre> </div></div> <p>They were all in an ll_ost_00_0?? thread with the same backtrace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>__ldiskfs_handle_dirty_metadata ldiskfs_free_inode ldiskfs_delete_inode generic_delete_inode generic_drop_inode iput osd_object_delete lu_object_free lu_object_put ofd_object_put ofd_destroy_by_fid ofd_destroy ofd_destroy ost_handle ptlrpc_server_handle_request ptlrpc_main </pre> </div></div> <p>The failures were on the secure network, so I can't upload logs.</p> LU-4382

kernel BUG at fs/jbd2/transaction.c:1033

Bug Blocker Resolved Fixed Zhenyu Xu Christopher Morrone mn4 Fri, 13 Dec 2013 02:33:44 +0000 Wed, 27 Aug 2014 17:27:42 +0000 Mon, 24 Feb 2014 22:05:00 +0000 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.5.1 0 10 <p>Bobijam</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>We had several more OSS nodes crash on this bug.</p> <p>it looks like the kernel stack is overflow but w/o further logs I could not know what code path took so much stack space.</p> <p>Could you elaborate on why you suspect a stack overflow? Just to make sure that we are on the same page, we are hitting the following J_ASSERT_JH in jbd2_journal_dirty_metadata():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 1026 if (jh-&gt;b_modified == 0) { 1027 /* 1028 * This buffer's got modified and becoming part 1029 * of the transaction. This needs to be done 1030 * once a transaction -bzzz 1031 */ 1032 jh-&gt;b_modified = 1; 1033 J_ASSERT_JH(jh, handle-&gt;h_buffer_credits &gt; 0); 1034 handle-&gt;h_buffer_credits--; 1035 } </pre> </div></div> <p>sorry, I meant journal credit deficiency.</p> <p>what condition is this hit? Client deleting files or OSS in recovery process.</p> <p>From the backtrace, it's in the ext4(ldiskfs) context which does not preserve enough credit for the inode delete operation.</p> <p>Is it possible that the OSS nodes do not have big enough journal devices for the bursting data change? </p> <blockquote><p>what condition is this hit? Client deleting files or OSS in recovery process.</p></blockquote> <p>I don't see any signs of recovery in the console logs. The assertion trips with nothing else in the log for more than an hour preceding it.</p> <blockquote><p>Is it possible that the OSS nodes do not have big enough journal devices for the bursting data change?</p></blockquote> <p>We are not using external journal devices. The OSS nodes have 15 OSTs, each device is aroun 3.3TB in size and around 80% full.</p> <p>OSS crashes are pretty frequent (every couple of days). We definitely need a plan to get this debugged.</p> <p>Hi bzzz,</p> <p>Can you enlighten me how to debug this transaction credit deficiency issue?</p> <p>I think Lustre isn't involved in this case - our transaction was closed in ofd_object_destroy() and later iput() was called on the inode.<br/> in ldiskfs_delete_inode():<br/> ...<br/> if (!ldiskfs_handle_has_enough_credits(handle, 3)) {<br/> err = ldiskfs_journal_extend(handle, 3);</p> <p>3 doesn't seem to be enough to remove inode from orphan list, free inode and adjust quota accounting ?</p> <p>Hi Alex,</p> <p>I checked the difference of ldiskfs code and vanilla code, in ldiskfs_free_inode()-&gt;ldiskfs_xattr_delete_inode(), we added delete_external_ea part which vanilla code does not contain, is it possible that we omitted count in the external EA inode unlink credit?</p> <p>master patch tracking at <a href="http://review.whamcloud.com/8881" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8881</a></p> <p>OSTs are dropping like flies around here lately.</p> <p>Since you have a theory now, is there perhaps a systemtap script or debugging patch that could be used to verify your theory? It would be reassuring to know that we are making progress on the right bug.</p> <p>The xattr_inode (large_xattr) feature shouldn't ever be used on an OST, so I don't think this can be the problem. That should only be used if there is an xattr larger than 4KB on the OST object, but we never store user xattrs or large layouts on the OST objects. The only xattrs we store there are the LMA (struct lov_mds_md) and fid (struct filter_fid) which should both fit in under 128 bytes. </p> <p>I would be more inclined to blame the truncate code, since this has traditionally been a source of problems because the size of a truncate transaction is potentially very large.</p> <p>Are quotas enabled on your filesystem (even just space accounting without enforcement)? Are the objects being unlinked particularly large?</p> <p>Yes quota accounting is enabled. It is always enabled in 2.4+, correct?</p> <p>I have no idea how large the objects might be. Some may certainly be large, for some definition of large.</p> <p>Alex,</p> <p>in ext4_free_inode()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* * Note: we must free any quota before locking the superblock, * as writing the quota to disk may need the lock as well. */ vfs_dq_init(inode); ext4_xattr_delete_inode(handle, inode); vfs_dq_free_inode(inode); vfs_dq_drop(inode); </pre> </div></div> <p>and vfs_dp_init() comment states</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/* It is better to call <span class="code-keyword">this</span> function outside of any transaction as it might * need a lot of space in journal <span class="code-keyword">for</span> dquot structure allocation. */ <span class="code-keyword">static</span> inline void vfs_dq_init(struct inode *inode) </pre> </div></div> <p>Is it possible that kernel code has not counted in the quota credit to be included in the 3 block credit?</p> <p>about truncate part, since truncate has happened before calculating remaining credit for inode free, so I'd think that truncate credit is contained safely.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (inode-&gt;i_blocks) ext4_truncate(inode); /* * ext4_ext_truncate() doesn't reserve any slop when it * restarts journal transactions; therefore there may not be * enough credits left in the handle to remove the inode from * the orphan list and set the dtime field. */ <span class="code-keyword">if</span> (!ext4_handle_has_enough_credits(handle, 3)) { err = ext4_journal_extend(handle, 3); <span class="code-keyword">if</span> (err &gt; 0) err = ext4_journal_restart(handle, 3); <span class="code-keyword">if</span> (err != 0) { ext4_warning(inode-&gt;i_sb, <span class="code-quote">"couldn't extend journal (err %d)"</span>, err); stop_handle: ext4_journal_stop(handle); sb_end_intwrite(inode-&gt;i_sb); <span class="code-keyword">goto</span> no_delete; } } </pre> </div></div> <p>iirc, to free an inode, we have to modify the following blocks: inode itself (dtime), orphan list header (sb), inode bitmap, group descriptor, then user accounting and group accounting = 6 blocks? and, iirc, we changed quota accounting to be atomic (so a part of transaction).</p> <p>Chris,<br/> Since it ends in a BUG(), is it possible that before to restart Lustre on the crashed OSS node you do some crash analysis to find concerned inode number and then extract its content using debugfs ?<br/> I would like to ensure that the issue is realy the xattr need for additional credits.</p> <p>I'm not sure why xattr is mentioned while 3 doesn't look enough with no xattrs even.</p> <p>Ok Alex, thanks to insist … I had to better read the full ticket's story. Talking with Johann it is also more clear for me what could be the specific scenario leading to this issue. Will try to post a patch proposal to ensure better computation of required credits, that will take care of quota-accounting if used and all the other blocks you mentionned.</p> <p>Chris, forget about my earlier requests of debugfs infos to determine xattr credits need, that do not apply (actually?) on OSSs.</p> <p>It would be a great exercise for somebody other than Alex to go through this callpath and count all possible blocks being journaled, list them all here for double verification and thus arrive to the updated reservation number.</p> <p>ext4_delete_inode()-&gt; ext4_free_inode(handle, inode)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> vfs_dq_init(inode); ext4_xattr_delete_inode(handle, inode); vfs_dq_free_inode(inode); vfs_dq_drop(inode); </pre> </div></div> <p>in vfs_dq_init()-&gt;ext4_dquot_initialize() (which is added in ldiskfs ext4-back-dquot-to patch)</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ext4_dquot_initialize</b></div><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> ext4_dquot_initialize(struct inode *inode, <span class="code-object">int</span> type) { handle_t *handle; <span class="code-object">int</span> ret, err; <span class="code-comment">/* We may create quota structure so we need to reserve enough blocks */</span> handle = ext4_journal_start(inode, 2*EXT4_QUOTA_INIT_BLOCKS(inode-&gt;i_sb)); <span class="code-keyword">if</span> (IS_ERR(handle)) <span class="code-keyword">return</span> PTR_ERR(handle); ret = dquot_initialize(inode, type); err = ext4_journal_stop(handle); <span class="code-keyword">if</span> (!ret) ret = err; <span class="code-keyword">return</span> ret; } </pre> </div></div> <p>in vfs_dq_free_inode()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* Dirtify all the dquots - <span class="code-keyword">this</span> can block when journalling */</span> <span class="code-keyword">for</span> (cnt = 0; cnt &lt; MAXQUOTAS; cnt++) <span class="code-keyword">if</span> (dquot[cnt]) mark_dquot_dirty(dquot[cnt]); </pre> </div></div> <p>and mark_dquot_dirty()-&gt;ext4_write_dquot()</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ext4_write_dquot</b></div><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> ext4_write_dquot(struct dquot *dquot) { <span class="code-object">int</span> ret, err; handle_t *handle; struct inode *inode; inode = dquot_to_inode(dquot); handle = ext4_journal_start(inode, EXT4_QUOTA_TRANS_BLOCKS(dquot-&gt;dq_sb)); <span class="code-keyword">if</span> (IS_ERR(handle)) <span class="code-keyword">return</span> PTR_ERR(handle); ret = dquot_commit(dquot); err = ext4_journal_stop(handle); <span class="code-keyword">if</span> (!ret) ret = err; <span class="code-keyword">return</span> ret; } </pre> </div></div> <p>and vfs_dq_drop()-&gt;ext4_dquot_drop()</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ext4_dquot_drop</b></div><div class="codeContent panelContent"> <pre class="code-java"> handle_t *handle; <span class="code-object">int</span> ret, err; <span class="code-comment">/* We may delete quota structure so we need to reserve enough blocks */</span> handle = ext4_journal_start(inode, 2*EXT4_QUOTA_DEL_BLOCKS(inode-&gt;i_sb)); <span class="code-keyword">if</span> (IS_ERR(handle)) { /* * We call dquot_drop() anyway to at least release references * to quota structures so that umount does not hang. */ dquot_drop(inode); <span class="code-keyword">return</span> PTR_ERR(handle); } ret = dquot_drop(inode); err = ext4_journal_stop(handle); <span class="code-keyword">if</span> (!ret) ret = err; <span class="code-keyword">return</span> ret; </pre> </div></div> <p>so for quota part, ext4_free_inode may misses 2*EXT4_QUOTA_INIT_BLOCKS(inode-&gt;i_sb) + 2*2 + 2*EXT4_QUOTA_DEL_BLOCKS(inode-&gt;i_sb) credits.</p> <p>the patch to verify the quota missed credit theory <a href="http://review.whamcloud.com/9187" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9187</a></p> <p>I don't think ext4_free_inode() needs to account for EXT4_QUOTA_INIT_BLOCK() because the user and group should already have quota allocated at wrote or chown time. I don't think it is possible to delete a file that has not already accounted in the quota. </p> <p>yes, I think you are right about the unnecessariness of the dquot initial credit counting.</p> <p>And further more, I checked dquot_drop()-&gt;dqput(), if the dquot drop need to release the quota structure, its operations also involves writing the dquot back to disk, so probably 2*EXT4_QUOTA_DEL_BLOCKS is enough for the whole quota operations.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>I find this global credits need evaluation very difficult to compute, regarding the number of nested handles/needs in underlying routines called and also depending on the current operation to be processed … So, I can understand why the 2*EXT4_QUOTA_INIT_BLOCK() credits need in vfs_dq_init()&gt;ext4_dquot_initialize()-&gt;dquot_initialize() could be ignored since it will not be used during a truncate, but then why do we also ignore the MAXQUOTAS*EXT4_QUOTA_TRANS_BLOCKS() (2*2 ?) for vfs_dq_free_inode()-&gt;mark_dquot_dirty()-&gt;ext4_write_dquot()-&gt;dquot_commit() ? And also, if the 1st reservation has been almost precise with this evaluation but more credits are required in ext4_delete_inode(), why do we need to extend with the same number ? </pre> </div></div> <p>&gt; And also, if the 1st reservation has been almost precise with this evaluation but more credits are required in <br/> ext4_delete_inode(), why do we need to extend with the same number ?</p> <p>because in some cases truncate can't fit a single transaction (this is true for other filesystems like ZFS as well), then it's hard to compute credits for truncate as it involves tree traversal (so we'd have to traverse twice).</p> <p>one way to "compute" is to extend handle_t with another counter and increment it in __ldiskfs_handle_dirty_metadata() or maintain some history. this way you can learn what code is involved, at least.</p> <p>Since OST does not use xattr inode, I've created another ticket for it (<a href="https://jira.whamcloud.com/browse/LU-4648" title="inode deletion transaction does not count large EA inode deletion credits" class="issue-link" data-issue-key="LU-4648"><del>LU-4648</del></a>).</p> <p>backport to b2_5:<br/> <a href="http://review.whamcloud.com/9334" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9334</a></p> <p>Landed for 2.6</p> Related LU-5392 Development Rank 1|hzwbbr: Rank (Obsolete) 12011 Severity