https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7001 Lustre <p>The problem unfortunately is not solved, even with the patch <a href="http://review.whamcloud.com/#/c/15841/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15841/</a><br/> I propose to raise the topic again <a href="https://jira.whamcloud.com/browse/LU-6944" title="LBUG: (osp_sync.c:1139:osp_sync_thread()) ASSERTION( thread-&gt;t_flags != SVC_RUNNING ) failed: 806 changes, 230 in progress, 7 in flight" class="issue-link" data-issue-key="LU-6944"><del>LU-6944</del></a><br/> The system restarts unexpectedly with errors<br/> Message from syslogd @ hard at Aug 13 8:56:38 ...<br/>   kernel: LustreError: 2796: 0: (osp_sync.c: 1139: osp_sync_thread ()) ASSERTION (thread-&gt; t_flags! = SVC_RUNNING) failed: 684 changes, 1137 in progress, 7 in flight<br/> Message from syslogd @ hard at Aug 13 8:56:38 ...<br/>   kernel: LustreError: 2796: 0: (osp_sync.c: 1139: osp_sync_thread ()) LBUG</p> <a href="https://build.hpdd.intel.com/job/lustre-reviews/34017/arch=x86_64,build_type=server,distro=el6.6,ib_stack=inkernel/">https://build.hpdd.intel.com/job/lustre-reviews/34017/arch=x86_64,build_type=server,distro=el6.6,ib_stack=inkernel/</a><br/> <a href="https://build.hpdd.intel.com/job/lustre-master/3137/arch=x86_64,build_type=server,distro=el6.6,ib_stack=inkernel/">https://build.hpdd.intel.com/job/lustre-master/3137/arch=x86_64,build_type=server,distro=el6.6,ib_stack=inkernel/</a> LU-7001

osp_sync.c: 1139: osp_sync_thread

Bug Critical Resolved Fixed Alexander Boyko Alex patch Thu, 13 Aug 2015 06:17:19 +0000 Tue, 18 Sep 2018 17:15:09 +0000 Wed, 13 Sep 2017 03:46:14 +0000 Lustre 2.11.0 Lustre 2.10.4 0 8 <p>Please provide the console logs with stack trace from the failing node. What operations are being done to trigger these errors?</p> <p>How to enable logs to provide them? Operation delete files</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/16335" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16335</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7001" title="osp_sync.c: 1139: osp_sync_thread" class="issue-link" data-issue-key="LU-7001"><del>LU-7001</del></a> osp: remove improper assert of sync thread<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ec59112060a4182b2c9b028ab38cf5f3dc3f8e8d</p> <p>We are hitting this issue repeatedly. I guess it will never recover unless we skip recovery or do something trick.</p> <p>Can we just remove the assertion? It seems this assertion is not proper, since the running thread has no idea when it will be requested to stop. Also, in osp_init0(), if ptlrpc_init_import() function returns a failure (ptlrpc_init_import() will not return any failure at least currently), it seems the assertion will fail. So this assertion looks dangerous.</p> <p>Finally, we walk around this problem by remove the CATALOGS file. I am wondering whether there is anyway to chack and recover broken llogs records...</p> <p>Li Xi, there are a couple of patches in flight that will repair or skip corrupted log records, but there may still be more types of corruption found on the future.</p> <p>Thank you Andreas for the information.</p> <p>Do you think it is possible to write a userspace tool to read as well as edit the llog files? I know that llog_reader is being changed, so hopefully, we will be able to at least dump the llog file. But since the llog files can be read locally from MDT/OST ldiskfs, maybe we can use a tool to remove wrong records mannually too?</p> <p>Alexander Boyko (alexander.boyko@seagate.com) uploaded a new patch: <a href="https://review.whamcloud.com/26132" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26132</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7001" title="osp_sync.c: 1139: osp_sync_thread" class="issue-link" data-issue-key="LU-7001"><del>LU-7001</del></a> osp: fix llog processing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bcebaa04977761773d24fed0821c44fcbd5bef83</p> <p>Alexander Boyko (alexander.boyko@seagate.com) uploaded a new patch: <a href="https://review.whamcloud.com/26359" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26359</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7001" title="osp_sync.c: 1139: osp_sync_thread" class="issue-link" data-issue-key="LU-7001"><del>LU-7001</del></a> tests: check osp_sync_thread for wrapped llog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8d779dab1025cd46ac44fc80f4dbd5ac85cba8a8</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26132/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26132/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7001" title="osp_sync.c: 1139: osp_sync_thread" class="issue-link" data-issue-key="LU-7001"><del>LU-7001</del></a> osp: fix llog processing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8da9fb0cf14cc79bf1985d144d0a201e136dfe51</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32097" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32097</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7001" title="osp_sync.c: 1139: osp_sync_thread" class="issue-link" data-issue-key="LU-7001"><del>LU-7001</del></a> osp: fix llog processing<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: bf6768549dfa09711daa66ccbf9db766c9f074f6</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/32097/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32097/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7001" title="osp_sync.c: 1139: osp_sync_thread" class="issue-link" data-issue-key="LU-7001"><del>LU-7001</del></a> osp: fix llog processing<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 10cc97e3c1487692b460702bf46220b1acb452ee</p> Related LU-6944 Development Rank 1|hzxkev: Rank (Obsolete) 9223372036854775807 Severity