https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7039 Lustre <p>Running tip of master with SWL and DNE. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2015-08-25 12:19:21 Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 2015-08-25 12:19:21 LustreError: 6378:0:(llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed: 2015-08-25 12:19:21 LustreError: 6384:0:(llog_osd.c:788:llog_osd_next_block()) lustre-MDT000b-osp-MDT0001: invalid llog tail at log id 0x3:2147484674/0 offset 3407872 2015-08-25 12:19:21 LustreError: 6384:0:(lod_dev.c:392:lod_sub_recovery_thread()) lustre-MDT000b-osp-MDT0001 getting update log failed: rc = -22 2015-08-25 12:19:21 LustreError: 6378:0:(llog_osd.c:778:llog_osd_next_block()) LBUG 2015-08-25 12:19:21 Pid: 6378, comm: lod0001_rec0005 2015-08-25 12:19:21 Aug 25 12:19:21 2015-08-25 12:19:21 iws12 kernel: LuCall Trace: 2015-08-25 12:19:21 streError: 6378: [&lt;ffffffffa04a2875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2015-08-25 12:19:21 0:(llog_osd.c:77 [&lt;ffffffffa04a2e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2015-08-25 12:19:21 8:llog_osd_next_ [&lt;ffffffffa08dad15&gt;] llog_osd_next_block+0xb75/0xbf0 [obdclass] 2015-08-25 12:19:21 block()) ASSERTI [&lt;ffffffffa08ccb4e&gt;] llog_process_thread+0xInitializing cgroup subsys cpuset 2015-08-25 12:19:21 Initializing cgroup subsys cpu </pre> </div></div> <p>Attempting to recreate and get a dump</p> Hyperion SWL test LU-7039

llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index ) failed:

Bug Blocker Resolved Fixed Di Wang Cliff White Tue, 25 Aug 2015 19:25:21 +0000 Mon, 8 Jun 2020 17:00:33 +0000 Wed, 27 Jan 2016 00:55:05 +0000 Lustre 2.8.0 Lustre 2.8.0 0 13 <p>Repeat, got the full stack this time:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2015-08-25 12:33:20 Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 2015-08-25 12:33:20 LustreError: 6140:0:(llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed: 2015-08-25 12:33:20 LustreError: 6134:0:(llog_osd.c:788:llog_osd_next_block()) lustre-MDT0005-osp-MDT0001: invalid llog tail at log id 0x3:1073742851/0 offset 3407872 2015-08-25 12:33:20 LustreError: 6134:0:(lod_dev.c:392:lod_sub_recovery_thread()) lustre-MDT0005-osp-MDT0001 getting update log failed: rc = -22 2015-08-25 12:33:20 LustreError: 6140:0:(llog_osd.c:778:llog_osd_next_block()) LBUG 2015-08-25 12:33:20 Pid: 6140, comm: lod0001_rec000b 2015-08-25 12:33:20 2015-08-25 12:33:20 Call Trace: 2015-08-25 12:33:20 [&lt;ffffffffa04a2875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2015-08-25 12:33:20 [&lt;ffffffffa04a2e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2015-08-25 12:33:20 [&lt;ffffffffa05f4d15&gt;] llog_osd_next_block+0xb75/0xbf0 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa05e6b4e&gt;] llog_process_thread+0x2de/0xfc0 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa05e78ed&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa12bc990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] 2015-08-25 12:33:20 [&lt;ffffffffa05eadb8&gt;] llog_cat_process_cb+0x458/0x600 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa05e71ba&gt;] llog_process_thread+0x94a/0xfc0 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa05e78ed&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa05ea960&gt;] ? llog_cat_process_cb+0x0/0x600 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa05e961d&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa12bc990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] 2015-08-25 12:33:20 [&lt;ffffffffa12bc990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] 2015-08-25 12:33:20 [&lt;ffffffffa04aebf1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2015-08-25 12:33:20 [&lt;ffffffffa05e9789&gt;] llog_cat_process+0x19/0x20 [obdclass] 2015-08-25 12:33:20 [&lt;ffffffffa12c1e4a&gt;] lod_sub_recovery_thread+0x69a/0xbc0 [lod] 2015-08-25 12:33:20 [&lt;ffffffffa12c17b0&gt;] ? lod_sub_recovery_thread+0x0/0xbc0 [lod] 2015-08-25 12:33:20 [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 2015-08-25 12:33:20 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 2015-08-25 12:33:20 [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 2015-08-25 12:33:20 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Lustre log attached</p> <p>Lustre log</p> <p>I saw this on some master failures from last night. I attributed the failures to <a href="https://jira.whamcloud.com/browse/LU-3263" title="llog_osd_next_block(): ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-3263"><del>LU-3263</del></a>. The assertions and stack traces look alike.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16084" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16084</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> llog: debug patches<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: de2d20f45a8e41a6e1510b596097e6c8ab270396</p> <p>Cliff: could you please try this debug patch and get the debug log(-1 level) for me? thanks.</p> <p>Per Cliff: we have tried the debug patch for a couple days and didn't get a debug log. Di is aware of this and will provide a new build.</p> <p>cliff: could you please try the test with this build <a href="http://review.whamcloud.com/#/c/13942/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13942/</a> , I added a few debug information and also some fixes, which are not landed yet. </p> <p>Same error happened during soak testing of build <a href="http://review.whamcloud.com/#/c/16685/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16685/</a><br/> (see <a href="https://wiki.hpdd.intel.com/pages/viewpage.action?spaceKey=Releases&amp;title=Soak+Testing+on+Lola#SoakTestingonLola-20150930" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/pages/viewpage.action?spaceKey=Releases&amp;title=Soak+Testing+on+Lola#SoakTestingonLola-20150930</a>)<br/> Here's the stack trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 1 20:06:27 lola-10 kernel: LustreError: 9078:0:(llog_osd.c:856:llog_osd_next_block()) ASSERTION( last_rec -&gt;lrh_index == tail-&gt;lrt_index ) failed: Oct 1 20:06:27 lola-10 kernel: LustreError: 9078:0:(llog_osd.c:856:llog_osd_next_block()) LBUG Oct 1 20:06:27 lola-10 kernel: Pid: 9078, comm: lod0004_rec0000 Oct 1 20:06:27 lola-10 kernel: Oct 1 20:06:27 lola-10 kernel: Call Trace: Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa05c1875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa05c1e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06d5b90&gt;] llog_osd_next_block+0xa40/0xc80 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06c6c0e&gt;] llog_process_thread+0x2de/0xfc0 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06c79ad&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa11ac990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06cae78&gt;] llog_cat_process_cb+0x458/0x600 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06c727a&gt;] llog_process_thread+0x94a/0xfc0 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06c79ad&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06caa20&gt;] ? llog_cat_process_cb+0x0/0x600 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06c96dd&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa11d8209&gt;] ? lod_sub_prep_llog+0x4f9/0x7a0 [lod] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa11ac990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa06c9849&gt;] llog_cat_process+0x19/0x20 [obdclass] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa11b1e2a&gt;] lod_sub_recovery_thread+0x69a/0xbc0 [lod] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffffa11b1790&gt;] ? lod_sub_recovery_thread+0x0/0xbc0 [lod] Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 Oct 1 20:06:27 lola-10 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Oct 1 20:06:27 lola-10 kernel: Oct 1 20:06:27 lola-10 kernel: LustreError: dumping log to /tmp/lustre-log.1443755187.9078 </pre> </div></div> <p>Also attached the debug log. I hope it contains some usable informations.</p> <p>debug log of MDS node lola-10</p> <p>The node continuously panics with same error at the moment. On thing to add for the test set-up. AS we hit LDEV-193 <br/> MDTs are mounted with option '<tt>abort_recovery</tt>'.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16714" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16714</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> llog: remove LASSERT check for record index<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f07a1980bd1e5cb08a00108ef286adc6bad4b4ea</p> <p>Frank, Cliff: I add some debug information in this patch. Could you please run test with this patch and collect -1 level debug log for me? Thank you.</p> <p>yep, we'll create a build containing your patch and the fix for LDEV-24. Otherwise this bug has no chance to occur.</p> <p>lola is updated with your patch, but MDSes crash on first mount.</p> <p>A single debug message beside a LBUG was written:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 5 09:51:39 lola-10 kernel: LustreError: 14865:0:(llog_osd.c:866:llog_osd_next_block()) soaked-MDT0000-osp -MDT0004: invalid llog head/tail at log id 0x2:165027/0 offset 163840 last_rec 1515870810/1515870810/151587081 0/1515870810tail 0/0 Oct 5 09:51:39 lola-10 kernel: LustreError: 14865:0:(llog_osd.c:868:llog_osd_next_block()) LBUG Oct 5 09:51:39 lola-10 kernel: Pid: 14865, comm: lod0004_rec0000 Oct 5 09:51:39 lola-10 kernel: Oct 5 09:51:39 lola-10 kernel: Call Trace: Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa05c1875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa05c1e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06d5c23&gt;] llog_osd_next_block+0xad3/0xd70 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06c6c0e&gt;] llog_process_thread+0x2de/0xfc0 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06c79ad&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa11ae990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06cae78&gt;] llog_cat_process_cb+0x458/0x600 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06c727a&gt;] llog_process_thread+0x94a/0xfc0 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06c79ad&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06caa20&gt;] ? llog_cat_process_cb+0x0/0x600 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06c96dd&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa11da549&gt;] ? lod_sub_prep_llog+0x4f9/0x7a0 [lod] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa11ae990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa06c9849&gt;] llog_cat_process+0x19/0x20 [obdclass] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa11b3e2a&gt;] lod_sub_recovery_thread+0x69a/0xbc0 [lod] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffffa11b3790&gt;] ? lod_sub_recovery_thread+0x0/0xbc0 [lod] Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 Oct 5 09:51:39 lola-10 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Oct 5 09:51:39 lola-10 kernel: </pre> </div></div> <p>According to the debug log </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000010:25.0:1444148492.324048:0:4548:0:(osp_trans.c:215:osp_update_request_destroy()) kfreed 'our': 72 at ffff880816a30740. 00000004:00000040:25.0:1444148492.324050:0:4548:0:(osp_md_object.c:1208:osp_md_read()) soaked-MDT0000-osp-MDT0004: total read [0x2000284a3:0x2:0x0] pos 163840 len 32768 00000004:00000001:25.0:1444148492.324059:0:4548:0:(osp_md_object.c:1209:osp_md_read()) Process leaving via out (rc=32768 : 32768 : 0x8000) 00000040:00020000:25.0:1444148492.324062:0:4548:0:(llog_osd.c:866:llog_osd_next_block()) soaked-MDT0000-osp-MDT0004: invalid llog head/tail at log id 0x2:165027/0 offset 163840 last_rec 1515870810/1515870810/1515870810/1515870810tail 0/0 </pre> </div></div> <p>tail-&gt;lrt_len = 0 &amp; tail-&gt;lrt_index = 0. Hmm, it seems update log is corrupted.</p> <p>Frank: could you please mount MDT0 as ldiskfs,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mount -t ldiskfs mdt0-dev /mnt/mds1 </pre> </div></div> <p>and get me this file /mnt/mds1/update_log_dir/<span class="error">&#91;0x2000284a3:0x2:0x0&#93;</span> and post it here. Thanks.</p> <p>Sigh, according the debug log and real llog update file, it seems llog file is corrupted, and the end of llog file is filled with junks. Unfortunately there are no enough information to know why it is corrupted. </p> <p>To fix the filesystem, it needs to mount all of MDTs as ldiskfs, and delete these files</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@testnode tests]# mount -t ldiskfs -o loop /tmp/lustre-mdtn /mnt/mdsn [root@testnode tests]# rm -rf /mnt/mdsn/update_log [root@testnode tests]# rm -rf /mnt/mdsn/update_log_dir [root@testnode tests]# umount /mnt/mdsn </pre> </div></div> <p>Then restart lustre. </p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16740" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16740</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> llog: skip to next chunk for corrupt record<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c612e7c78344b2ff88c0d5803b558c9991ab766f</p> <p>This is not a real fix. Just fix log_reader to make it skip the corrupt records.</p> <p>the same issue with <a href="https://jira.whamcloud.com/browse/LU-7100" title="conf-sanity test_84 LBUGS with “(llog_osd.c:811:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index )”" class="issue-link" data-issue-key="LU-7100"><del>LU-7100</del></a>, but different llog</p> <p>Di, I notice that changes made by you some time ago:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lgi-&gt;lgi_off = max_t(__u64, lgi-&gt;lgi_attr.la_size, lgi-&gt;lgi_off); </pre> </div></div> <p>Not sure why that was made, could you explain that a bit? IIRC, that was done so because la_size may be not yet updated after first write, e.g. header, right?<br/> If so then the same can happen with pad record, consider that pad was written, the lgi_off with offset is lost after that - it is rewriten by 0 for header update, but la_size may be not yet updated, so the record itself will overwrite the pad record.</p> <p>The solution could be saving the lgi_off after pad writting in local variable and use it.</p> <p>I've just cook the patch if that make sense: <a href="http://review.whamcloud.com/16771" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16771</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>If so then the same can happen with pad record, consider that pad was written, the lgi_off with offset is lost after that - it is rewriten by 0 for header update, but la_size may be not yet updated, so the record itself will overwrite the pad record. </pre> </div></div> <p>Hmm, we suppose to update the size in osp cache, see osp_md_write(). Note: this is only for remote llog object, for local llog object, adding pad record will update la_size in time, since you also see this problem in <a href="https://jira.whamcloud.com/browse/LU-7100" title="conf-sanity test_84 LBUGS with “(llog_osd.c:811:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index )”" class="issue-link" data-issue-key="LU-7100"><del>LU-7100</del></a> for local log, so I doubt this is the reason.</p> <p>If so, then we don't need max_t(__u64, lgi-&gt;lgi_attr.la_size, lgi-&gt;lgi_off); anymore, right? it can be just la_size always.</p> <p>As I can see that obj-&gt;ooa may not exists, and it seems in most cases that is so, because it it is created just upon create of file what is not case with llog. Nevertheless, I agree that my suggestion doesn't explain corruption in local llog, I've added some debug in patch, let's wait for new issues.</p> <p>Both patches installed for this bug + LDEV-24 patch that landed after creation of the build (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20151007" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20151007</a>) and the cleanup procedure was executed.<br/> The error didn't happened since now.<br/> The following error message was printed during failover of MDT-0004 from <tt>lola-10</tt> --&gt; <tt>lola-11</tt> and on other MDSes, too:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 8 16:23:23 lola-11 kernel: LDISKFS-fs (dm-6): recovery complete Oct 8 16:23:23 lola-11 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. quota=on. Opts: Oct 8 16:23:23 lola-11 kernel: Lustre: soaked-MDT0006: Connection restored to 192.168.1.111@o2ib10 (at 0@lo) Oct 8 16:23:23 lola-11 kernel: Lustre: Skipped 3 previous similar messages Oct 8 16:23:23 lola-11 kernel: LustreError: 11-0: soaked-MDT0003-osp-MDT0005: operation mds_connect to node 1 92.168.1.109@o2ib10 failed: rc = -114 Oct 8 16:23:23 lola-11 kernel: LustreError: Skipped 2 previous similar messages Oct 8 16:23:23 lola-11 kernel: Lustre: soaked-MDT0005: Imperative Recovery enabled, recovery window shrunk fr om 300-900 down to 150-450 Oct 8 16:23:23 lola-11 sshd[5890]: Received disconnect from 10.4.0.116: 11: disconnected by user Oct 8 16:23:23 lola-11 sshd[5890]: pam_unix(sshd:session): session closed for user root Oct 8 16:23:23 lola-11 kernel: LustreError: 5981:0:(llog.c:534:llog_process_thread()) soaked-MDT0006-osp-MDT0005: Invalid record: index 8 but expected 7 Oct 8 16:23:23 lola-11 kernel: LustreError: 5981:0:(lod_dev.c:392:lod_sub_recovery_thread()) soaked-MDT0006-osp-MDT0005 getting update log failed: rc = -34 </pre> </div></div> <p>Not sure whether it is related to this bug.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16797" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16797</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> tgt: Delete txn_callback correctly in tgt_init()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 57daeb1473bbf2cf3c3cfde83890818beee55f28</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16839" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16839</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> osp: set update req error once import invalid<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 946fbe320ab6f093885247ec8234f2e4a28b50dc</p> <p>soak ran with build for CentOS-6.6 of build associated with change <a href="http://review.whamcloud.com/#/c/16838/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16838/</a><br/> After reset of single MDS and remount the error occurred again:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 15 16:08:03 lola-10 kernel: LDISKFS-fs warning (device dm-5): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Oct 15 16:08:03 lola-10 kernel: Oct 15 16:09:00 lola-10 kernel: LDISKFS-fs (dm-5): recovery complete Oct 15 16:09:00 lola-10 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. quota=on. Opts: Oct 15 16:09:01 lola-10 kernel: LustreError: 137-5: soaked-MDT0004_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Oct 15 16:09:01 lola-10 kernel: Lustre: soaked-MDT0005: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 Oct 15 16:09:01 lola-10 kernel: Lustre: 4390:0:(llog.c:520:llog_process_thread()) invalid length 0 in llog record for index 0/43 Oct 15 16:09:01 lola-10 kernel: LustreError: 4390:0:(lod_dev.c:402:lod_sub_recovery_thread()) soaked-MDT0002-osp-MDT0005 getting update log failed: rc = -22 Oct 15 16:09:01 lola-10 kernel: LustreError: 4242:0:(mdt_handler.c:5596:mdt_iocontrol()) soaked-MDT0005: Aborting recovery for device Oct 15 16:09:01 lola-10 kernel: LustreError: 4242:0:(ldlm_lib.c:2473:target_stop_recovery_thread()) soaked-MDT0005: Aborting recovery Oct 15 16:09:01 lola-10 kernel: Lustre: 4395:0:(ldlm_lib.c:1946:target_recovery_overseer()) recovery is aborted, evict exports in recovery Oct 15 16:09:01 lola-10 kernel: Lustre: soaked-MDT0005: disconnecting 16 stale clients Oct 15 16:09:07 lola-10 kernel: Lustre: soaked-MDT0005: Denying connection for new client e20f50f6-1966-798c-abf7-2d3bde20760a(at 192.168.1.126@o2ib100), waiting for 16 known clients (0 recovered, 0 in progress, and 16 evicted) to recover in 21116921:47 Oct 15 16:09:08 lola-10 kernel: LustreError: 4393:0:(llog_osd.c:856:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed: Oct 15 16:09:08 lola-10 kernel: LustreError: 4393:0:(llog_osd.c:856:llog_osd_next_block()) LBUG Oct 15 16:09:08 lola-10 kernel: Pid: 4393, comm: lod0005_rec0006 Oct 15 16:09:08 lola-10 kernel: Oct 15 16:09:08 lola-10 kernel: Call Trace: Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa05c1875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa05c1e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06d5bab&gt;] llog_osd_next_block+0xa4b/0xc90 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06c6c2e&gt;] llog_process_thread+0x2de/0xfc0 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06c79cd&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa11a7990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06cae98&gt;] llog_cat_process_cb+0x458/0x600 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06c729a&gt;] llog_process_thread+0x94a/0xfc0 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa07168b4&gt;] ? dt_read+0x14/0x50 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06c79cd&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06caa40&gt;] ? llog_cat_process_cb+0x0/0x600 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06c96fd&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa11d3589&gt;] ? lod_sub_prep_llog+0x4f9/0x7a0 [lod] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa11a7990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa06c9869&gt;] llog_cat_process+0x19/0x20 [obdclass] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa11a8c8e&gt;] lod_sub_recovery_thread+0x26e/0xb90 [lod] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffffa11a8a20&gt;] ? lod_sub_recovery_thread+0x0/0xb90 [lod] Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 Oct 15 16:09:08 lola-10 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Oct 15 16:09:08 lola-10 kernel: </pre> </div></div> <p>I just figured out a way to reproduce the problem, and also prove my thought that the failure of llog records cause the corruption. I will update the patch.</p> <p>I update the fix along with other DNE patches to <a href="http://review.whamcloud.com/16838" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16838</a> . After the test approved this actually works, I will update 16389.</p> <p>Di, could you describe what caused llog corruption? Was it wrong usage of llog by external caller or problem inside llog itself?</p> <p>Please check the test case replay single 117 in patch 16838. Basically it because of the llog header is not protected by lock when used by OSP for update log. Basically, it happens like this</p> <p>1. For example OSP adds 3 updates into the OSP sending list. All of them will write to the same remote llog file. So they will share the same bitmap and log index etc. Though the bitmap in 2nd will include 1st's bit, and the 3rd includes both 1st and 2nd bits.</p> <p>2. And the 1 updates failed for some reason, which might happen especially when we restart 2 MDTs at the same time in soak-test. </p> <p>3. After recovery, it will keep writing 2nd and 3 rd updates, which might include 1st bitmap, but there are no 1st record because of the failure of step 2.</p> <p>So right now, I will let all of updates in the sending list fail once this happen, then refresh the header and also llog size for the new RPC.</p> <p>That said this probably not the reason for local log corruption. And another thing I do not understand is that why this just start to happen recently (since last 2 weeks)? I checked the recent commit for llog, and did not find any suspicious patch? Do you have any clue?</p> <p>I don't see how the dt_attr_get() synchronized with dt_write() over OSP. As I can see there is no cache for attributes if llog wasn't created but opened. That means the dt_attr_get() will cause RPC to another server, but dt_write() might not be even applied remotely, so returned size might be smaller and llog will be corrupted as result. Is that true or I miss something?</p> <p>the llog update (write + size update) is serialized by rpc_version, see osp_send_update_thread(). </p> <p>It is not about updates but about attr_get() which is not serialized with updates, so can return smaller size</p> <p>They are serialize by lgh_lock in llog_cat_add_rec(). </p> <p>I don't see how lgh_lock helps here. Consider the following scenario:</p> <ul class="alternate" type="square"> <li>llog_osd_write() issues new record, write RPC is scheduled for another server</li> <li>new llog_osd_write() is called immediately and issues attr_get() RPC for the same llog to the remote server</li> <li>since these two RPCs are not serialized and were issued almost together technically the attr_get() RPC might return size value before write will be applied on remote server</li> </ul> <p>This is not the case if llog write RPC is synchronous but it is not so, isn't it?</p> <p>Hmm, Mike, I think you are right. We need update the size for the OSP object after write.</p> <p>Do you mean to make opo_ooa exists all time? That would solve this problem I think.</p> <p>Yes, I believe so. I added this fix to 16838 (along with a few other changes) and soak-test will try to see if this can resolve the corrupt issue. </p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16969" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16969</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> llog: update llog header and size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9191d9a7f79c739d71dc7652333b9f07456218ad</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16797/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16797/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> tgt: Delete txn_callback correctly in tgt_init()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 44e9ec0b46fc46cc72bebbdc35e4a59a0397a81c</p> <p>Di: For build '20151027' (<a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20151027" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20151027</a>)<br/> which doesn't include change 16797, as far as I can see, the problem is still present:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 28 07:10:59 lola-8 kernel: LustreError: 6954:0:(llog_osd.c:874:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed: Oct 28 07:10:59 lola-8 kernel: LustreError: 6954:0:(llog_osd.c:874:llog_osd_next_block()) LBUG Oct 28 07:10:59 lola-8 kernel: Pid: 6954, comm: lod0003_rec0007 Oct 28 07:10:59 lola-8 kernel: Oct 28 07:10:59 lola-8 kernel: Call Trace: Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa07fc875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa07fce77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa0917a7b&gt;] llog_osd_next_block+0xa4b/0xc90 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffff81174450&gt;] ? cache_alloc_refill+0x1c0/0x240 Oct 28 07:10:59 lola-8 kernel: LustreError: 6948:0:(llog.c:534:llog_process_thread()) soaked-MDT0000-osp-MDT0003: Invalid re cord: index 9421 but expected 9420 Oct 28 07:10:59 lola-8 kernel: LustreError: 6948:0:(lod_dev.c:402:lod_sub_recovery_thread()) soaked-MDT0000-osp-MDT0003 gett ing update log failed: rc = -34 Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa0906d3e&gt;] llog_process_thread+0x2de/0xfc0 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa0904d5c&gt;] ? llog_init_handle+0x11c/0x950 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa0907add&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa1374990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa090b108&gt;] llog_cat_process_cb+0x458/0x600 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa09073aa&gt;] llog_process_thread+0x94a/0xfc0 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa0907add&gt;] llog_process_or_fork+0xbd/0x5d0 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa090acb0&gt;] ? llog_cat_process_cb+0x0/0x600 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa090996d&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa13a0589&gt;] ? lod_sub_prep_llog+0x4f9/0x7a0 [lod] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa1374990&gt;] ? lod_process_recovery_updates+0x0/0x420 [lod] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa0909ad9&gt;] llog_cat_process+0x19/0x20 [obdclass] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa1375c8e&gt;] lod_sub_recovery_thread+0x26e/0xb90 [lod] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffffa1375a20&gt;] ? lod_sub_recovery_thread+0x0/0xb90 [lod] Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Oct 28 07:10:59 lola-8 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 </pre> </div></div> <p>Frank: I updated <a href="http://review.whamcloud.com/16838" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16838</a>, and added more debug information there. Could you please retry with the patch. Thanks.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16740/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16740/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> llog: skip to next chunk for corrupt record<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 04f4023cf59b6e5a1634ba492cd813dcb1af0c7c</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17199" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17199</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> recovery: abort update recovery once fails<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 70905df1d7ea16d50c927b6af9957bced89a0f3b</p> <p>Just update, it seems corruption disappears in the build of 20151120, though we need run more test to confirm this. Currently the soak-test is blocked by <a href="https://jira.whamcloud.com/browse/LU-7456" title="kernel panic on ldiskfs_journal_commit_callback+0x4a/0x80 " class="issue-link" data-issue-key="LU-7456"><del>LU-7456</del></a>, and we will continue soak-test to check this problem once 7456 is fixed.</p> <p>Just reminder, <a href="http://review.whamcloud.com/16969" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16969</a> and <a href="http://review.whamcloud.com/17199" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17199</a> are key fixes for this problem.</p> <p>The error below happens during soak testing of change 16838 patch set #31 (no Wiki entry for build exits, yet) on cluster <em>lola</em>. DNE is enabled and MDSes are configured in active-active HA failover configuration. </p> <ul> <li>Primary resources of MDT <tt>lola-11</tt> were failed back at Dec, 3 20:18.<br/> The allocation of slabs increased continuously till ~ 31 GB till crash </li> <li>MDS node <tt>lola-11</tt> crashed with oom-killer at Dec, 4 00:21 (local time). (see also <a href="https://jira.whamcloud.com/browse/LU-7432" title="oom-killer started on MDSes" class="issue-link" data-issue-key="LU-7432"><del>LU-7432</del></a>)</li> <li><tt>ptlrpc_cache</tt> seems to be the biggest consumer<br/> Attached <tt>lola-11</tt>'s messages, console log, vmcore-dmesg file, collectl (version V4.0.2-1) files (for time interval specified above). Also<br/> attached files containing extracted counters for memory, slab totals and per slab allocation.</li> </ul> <p>The crash dump has been saved to <tt>lola-1:/scratch/crashdumps/lu-7039/127.0.0.1-2015-12-04-00\:22\:36</tt>.</p> <p>It turned out that the <tt>collectl</tt> raw files are to big to be uploaded to Jira. I saved them to <tt>lola-1:/scratch/crashdumps/lu-7039</tt>.</p> <p>This OOM is not caused by llog corruption, so it is a new problem (<a href="https://jira.whamcloud.com/browse/LU-7517" title="oom killer active after failback of MDS resources" class="issue-link" data-issue-key="LU-7517"><del>LU-7517</del></a>).</p> <p>another instance on master DNE mode<br/> <a href="https://testing.hpdd.intel.com/test_sets/b57ce146-bbfd-11e5-8506-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b57ce146-bbfd-11e5-8506-5254006e85c2</a><br/> client and server: lustre-master build#3305</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16969/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16969/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a> llog: update llog header and size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e1745ed18d8e28f3cf3d72df3b7ef50d83f36601</p> <p>Landed for 2.8.0</p> Duplicate LU-7391 LU-7100 Related LU-6831 LU-7715 LU-6994 LU-7455 Development End date Fri, 4 Dec 2015 19:25:21 +0000 Rank 1|hzxlbb: Rank (Obsolete) 9223372036854775807 Severity Start date Tue, 25 Aug 2015 19:25:21 +0000