https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6556 Lustre <p>After our last lustre upgrade, On tera100 and tgcc site, some <br/> lustre fs have meet the same corruption on the changelog_catalog <br/> The robinhood node panic like in the <a href="https://jira.whamcloud.com/browse/LU-6471" title="Unexpected Lustre Client LBUG in llog_write()" class="issue-link" data-issue-key="LU-6471"><del>LU-6471</del></a> and the crash analyze <br/> show that is the changelog-catalog file that have a corruption. <br/> The file is too big than the maximum size of this type of file and <br/> the record who produces the panic is not in the right place.</p> redhat kernel 2.6.32_504.8.1.el6<br/> lustre 2;5.3.90 + <a href="https://jira.whamcloud.com/browse/LU-5740" title="Kernel upgrade [RHEL6.6 2.6.32-504.el6]" class="issue-link" data-issue-key="LU-5740"><strike>LU-5740</strike></a> <a href="https://jira.whamcloud.com/browse/LU-4582" title="After failing over Lustre MGS node to the secondary, client mount fails with -5" class="issue-link" data-issue-key="LU-4582"><strike>LU-4582</strike></a> <a href="https://jira.whamcloud.com/browse/LU-5678" title="kernel crash due to NULL pointer dereference in kiblnd_pool_alloc_node()" class="issue-link" data-issue-key="LU-5678"><strike>LU-5678</strike></a> <a href="https://jira.whamcloud.com/browse/LU-5393" title="LBUG: (ost_handler.c:882:ost_brw_read()) ASSERTION( local_nb[i].rc == 0 ) failed" class="issue-link" data-issue-key="LU-5393"><strike>LU-5393</strike></a> <a href="https://jira.whamcloud.com/browse/LU-3727" title="LBUG (llite_nfs.c:281:ll_get_parent()) ASSERTION(body-&gt;valid &amp; OBD_MD_FLID) failed" class="issue-link" data-issue-key="LU-3727"><strike>LU-3727</strike></a> <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><strike>LU-4528</strike></a> <a href="https://jira.whamcloud.com/browse/LU-5522" title="ofd_prolong_extent_locks()) ASSERTION( lock-&gt;l_flags &amp; 0x0000000000000020ULL ) failed" class="issue-link" data-issue-key="LU-5522"><strike>LU-5522</strike></a> <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><strike>LU-5264</strike></a> <a href="https://jira.whamcloud.com/browse/LU-6049" title="General Protection Fault at echo_session_key_fini+0xa9" class="issue-link" data-issue-key="LU-6049"><strike>LU-6049</strike></a> <a href="https://jira.whamcloud.com/browse/LU-6084" title="Tests are failed due to &#39;recovery is aborted by hard timeout&#39;" class="issue-link" data-issue-key="LU-6084"><strike>LU-6084</strike></a> <a href="https://jira.whamcloud.com/browse/LU-5764" title="Crash of MDS on &quot;apparent buffer overflow&quot;" class="issue-link" data-issue-key="LU-5764"><strike>LU-5764</strike></a> LU-6556

changelog catalog corruption if all possible records is define

Bug Critical Resolved Fixed Bruno Faccini Antoine Percher Sat, 2 May 2015 15:46:04 +0000 Thu, 7 Jan 2016 05:43:11 +0000 Wed, 28 Oct 2015 18:30:01 +0000 Lustre 2.5.3 Lustre 2.8.0 0 18 <p>Bruno </p> <p>Can you please confirm whether this does indeed meet the profile of <a href="https://jira.whamcloud.com/browse/LU-6471" title="Unexpected Lustre Client LBUG in llog_write()" class="issue-link" data-issue-key="LU-6471"><del>LU-6471</del></a>?</p> <p>Thanks</p> <p>Peter</p> <p>Assigning to me since I have been working on this issue when being on-site with Antoine.</p> <p>In fact, after our joint analysis, it seems that the crash has occurred a few time after the upgrade+reboot of all nodes because of the 2 combined things :<br/> 1) a ChangeLog Catalog that has already looped-back<br/> 2) integrating/running with <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> patch that seems to have introduced a regression where looped-back Catalogs are not handled correctly and are only expected to grow when it is not the case.</p> <p>Will push a patch soon to master where the problem seems to be also present (but still undetected due to the need of a looped-back Catalog situation to trigger which should not occur so frequently ...).</p> <p>Peter, no in this particular case the similar LBUG than for <a href="https://jira.whamcloud.com/browse/LU-6471" title="Unexpected Lustre Client LBUG in llog_write()" class="issue-link" data-issue-key="LU-6471"><del>LU-6471</del></a> is valid since it is triggered due to the ChangeLog Catalog corruption I just explained before in my previous comment.</p> <p>Please find in attachment customer trace analyze file (log_lu-6556_b.txt)<br/> to understand the changelog-catalog coruption </p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14912" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14912</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a> obdclass: re-allow catalog loopback<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8112eb1a2d3dd5beed00c75d76295ae1362ab19a</p> <p>I have created <a href="https://jira.whamcloud.com/browse/LU-6634" title="(osd_handler.c:901:osd_trans_start()) ASSERTION( get_current()-&gt;journal_info == ((void *)0) ) failed: when reaching Catalog full condition" class="issue-link" data-issue-key="LU-6634"><del>LU-6634</del></a> because during my testing of my patch for this/<a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a> ticket, I wanted to check what happen (expected ENOSPC return!) after Catalog has looped-back and fills up, but I got a "(osd_handler.c:901:osd_trans_start()) ASSERTION( get_current()-&gt;journal_info == ((void *)0) ) failed:" LBUG. I have identified this to be caused by the fact that in the error path for llog_cat_new_log(), llog_destroy() is called to destroy the new plain LLOG for which the reference can't be recorded into Catalog because there is no slot available to do so, and this triggers the Assertion because there is already a started transaction from llog_cat_add(), when llog_destroy() wants to start its own transaction.</p> <p>I think the other thing that is needed here is to automatically unregister ChangeLog watcher(s) if the changelog is full or the MDS runs out of space (by default), or block all MDS operations until the ChangeLog can be written (if /proc tunable is set to make ChangeLog updates mandatory). It should unregister starting with the oldest watcher on the assumption that the older watcher was forgotten and newer ones are still running, and that this will release the most space. The unregistration should cancel records up to the next watcher, or all remaining records if no other watchers are left.</p> <p>I suggest going a step further and proactively remove stale watchers after a configurable period or when hitting a max watermark to try o avoid running out of space. Also, being unregistered is a reasonable notification to the application that they've lost their changelog feed and need to resync. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14912/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14912/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a> obdclass: re-allow catalog to wrap around<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4691290f6d39bffaa3e463697fbc3ac351015e76</p> <p>Andreas, Robert,<br/> I also think that your concerns are really good points for more ChangeLogs related enhancements, but also that they should addressed in a separate ticket, when this ticket could now be closed.<br/> Do you agree ?</p> <p>Bruno, that is fine. Please file a separate bug and copy over relevant comments before closing this one, so that they are not forgotten.</p> <p><a href="https://jira.whamcloud.com/browse/LU-7340" title="ChangeLogs catalog full condition should be handled more gracefully" class="issue-link" data-issue-key="LU-7340"><del>LU-7340</del></a> has been created to address previous ChangeLogs related and more graceful handling of ENOSPC conditions.</p> <p>Landed to 2.8</p> <p>Backported of the <a href="http://review.whamcloud.com/14912" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14912</a> patch also need the patch <a href="http://review.whamcloud.com/17052" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17052</a> "<a href="https://jira.whamcloud.com/browse/LU-7329" title="sanity test_60a timeouts with “* invoking oom-killer” " class="issue-link" data-issue-key="LU-7329"><del>LU-7329</del></a> obdclass: sync device to flush journal callbacks" patch to avoid introducing test failures in sanity test_60a. </p> Related LU-4528 LU-7329 LU-7138 LU-6634 LU-7241 LU-6954 Development Rank 1|hzxcdz: Rank (Obsolete) 9223372036854775807 Severity