https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6954 Lustre <p>lustre-2.5.4-4chaos_2.6.32_504.16.2.1chaos.ch5.3.x86_64.x86_64</p> <p>The mds service on both porter and stout fails to start. Able to import zfs pool on both systems with no problem. The mgs device mounts with no problem but the mdt on both systems fails to mount. Doing a "writeconf" on the stout mds did not help. The following console messages were reported on stout-mds1 console: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2015-08-02 16:38:26 Lustre: Lustre: Build Version: 2.5.4-4chaos-4chaos--PRISTINE-2.6.32-504.16.2.1chaos.ch5.3.x86_64 2015-08-02 16:38:27 Lustre: MGC172.21.1.99@o2ib200: Connection restored to MGS (at 0@lo) 2015-08-02 16:38:28 Lustre: MGS: Logs for fs fsrzb were removed by user request. All servers must be restarted in order to regenerate the logs. 2015-08-02 16:38:30 LustreError: 11-0: fsrzb-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2015-08-02 16:38:31 Lustre: 12934:0:(llog_cat.c:718:llog_cat_reverse_process()) catalog 0x2:10 crosses index zero 2015-08-02 16:38:31 LustreError: 12934:0:(mdd_device.c:305:mdd_changelog_llog_init()) fsrzb-MDD0000: changelog init failed: rc = -5 2015-08-02 16:38:31 LustreError: 12934:0:(mdd_device.c:380:mdd_changelog_init()) fsrzb-MDD0000: changelog setup during init failed: rc = -5 2015-08-02 16:38:31 LustreError: 12934:0:(mdd_device.c:963:mdd_prepare()) fsrzb-MDD0000: failed to initialize changelog: rc = -5 2015-08-02 16:38:31 Lustre: fsrzb-MDT0000: Unable to start target: -5 2015-08-02 16:38:31 Lustre: Failing over fsrzb-MDT0000 2015-08-02 16:38:32 Lustre: server umount fsrzb-MDT0000 complete 2015-08-02 16:38:32 LustreError: 12934:0:(obd_mount.c:1331:lustre_fill_super()) Unable to mount (-5) </pre> </div></div> <p>A workaround was found to allow the MDT to mount:<br/> Mounting the MDT via ZPL<br/> Delete the changelog_catalog and changelog_users files<br/> Unmount<br/> Mount the MDT via lustre in the normal manner</p> lustre-2.5.4-4chaos_2.6.32_504.16.2.1chaos.ch5.3.x86_64.x86_64 LU-6954

LustreError: 12934:0:(mdd_device.c:305:mdd_changelog_llog_init()) fsrzb-MDD0000: changelog init failed: rc = -5

Bug Critical Closed Fixed Bruno Faccini Olaf Faaland llnl Tue, 4 Aug 2015 17:09:54 +0000 Fri, 19 Feb 2016 19:10:57 +0000 Fri, 19 Feb 2016 19:10:57 +0000 Lustre 2.5.4 0 6 <p>Hello Olaf,<br/> Did you keep a copy of changelog_catalog and changelog_users files that you can provide ?</p> <p>Bruno,<br/> Sorry, yes, attached now.</p> <p>Thanks Olaf, and here is what I can tell after analyzing the changelog_catalog file you have provided.<br/> I am not able to confirm that Lustre v2.5.4, you run with, contains or not patch from <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> (<a href="http://review.whamcloud.com/#/c/10108/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10108/</a>, Commit 7c243a561ffe8503a6abf5c4cafef0c3566192bc). Can you check this for me?<br/> But if this is the case and since your changelog_catalog had just reached its end and was about to loop-back, I think you likely encountered the same kind of regression described in <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a>.</p> <p>I find it exceptionally unlikely that two different filesystems had both independently "just reached" the end of the changelog_catalog and were just about to loop back at exactly the same time. More explanation is needed.</p> <p>Hello Chris, thanks to warn me about the fact that 2 FSs are affected, I should have better read the description text for this ticket, sorry about that.</p> <p>But, if the same symptoms/msgs have occurred for both Filesystems failures, I can already confirm that the "crosses index zero" msg is an indication of a Catalog loop-back, and I will also need the 2nd changelog_catalog file for the 2nd filesystem to analyze it.<br/> We may have end up in a situation where both Catalog have loop-back, when only one/1st was just doing so ... And this since last Filesystems restarts.</p> <p>And BTW, I have double-checked the 1st Catalog you have provided and I can also confirm you that it shows the same corruption (Catalog records written past normal end, leading to a Catalog size &gt; header+bitmap+records) than what has been found for <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a>.</p> <p>Concerning the fact I used the "just reached" comment, this may come from the fact that, for an unexplained reason at the moment, bits at the beginning of the bitmap have been cleared (or may be never set).</p> <p>Bruno,</p> <p>I confirmed that both filesystems produced the same sequence of error messages when attempting to start the MDT, including the "crosses index zero" and "changelog init failed" messages, same rc's.</p> <p>We do have the <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> patch in our build.</p> <p>I'll attach the second changelog_catalog. The one you've already seen is from porter.</p> <p>Bruno,</p> <p>Do you have any updates on this? I see that the stout catalog file contains 67272 records, and it looks like the bitmap has only 64767 bits for tracking the status of the non- llog_log_hdr records. So it does seem to me that the changelog_catalog file is corrupt.</p> <p>The records that appear after that have indices in the range 12197 - 14701, which seems odd. The code in llog_osd_prev_block() appears to me to assume that the records within a block have monotonically increasing indices, since only lrt_index is generally checked before deciding whether to read another block from disk or not. Am I correct that requirement for increasing indices?</p> <p>thanks,<br/> Olaf</p> <p>Olaf,<br/> Thanks to have provided the second ChangeLog catalog file for stout.<br/> You are correct in the analysis of his already looped-back and corrupted content, which is again the same than for porter, and the one that has already been investigated in <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a>.</p> <p>So the scenario which is likely to have happen for both filesystems is that prior to start running with patch for <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> in, both ChangeLog catalogs had already looped-back, but the patch has caused new records to be put at the end of each files instead of updating their corresponding records in-place. The corruption was detected at next FSs restart.</p> <p>Am I right when I suspect that problem has occured during the 2nd restart for each FS since upgrading with <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> in ?</p> <p>Bruno,</p> <p>Yes, it is likely we encountered this error on the 2nd restart after upgrading with <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a>.</p> <p>I am looking at the code in llog_osd_write_rec() and your patch for <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a>. I see that in the old code, lgi_off is set to la_size unconditionally.</p> <p>thanks,<br/> Olaf </p> <p>Bruno,</p> <p>I can see how we could end up with the changelog_catalog file corruption if, before we upgraded to <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> code, our changelog_catalog was already wrapped around, so that lgh_last_idx == 12196 and changelog_catalog size == 4,153,280. I think this is what you are saying happened.</p> <p>However, in the <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> patch, and in the previous code it applied to, I don't see something implementing changelog_catalog wrap-around - setting lgh_last_idx in some way other than incrementing or setting to 0 when creating changelog_catalog for the first time. Do you?</p> <p>thanks,<br/> Olaf</p> <p>I guess what I'm really asking is, did changelog_catalogs wrap around prior to the <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> patch? Some comment made me think so, maybe I misunderstood.</p> <p>thanks,<br/> Olaf</p> <p>Bruno,</p> <p>Looks to me like this code in llog_cat_new_log() implemented wrap-around before <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a>. Please confirm I'm not misreading.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">76 bitmap_size = LLOG_BITMAP_SIZE(llh); 77 78 index = (cathandle-&gt;lgh_last_idx + 1) % bitmap_size; ... 118 cathandle-&gt;lgh_last_idx = index; 119 llh-&gt;llh_tail.lrt_index = index; </pre> </div></div> <p>thanks,<br/> Olaf</p> <p>Olaf,<br/> To be quick, Catalog wrap-around was working before <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> patch and no longer after, and also situations where Catalog has already wrapped-around will lead to corruption with new records being written past of expected Catalog's normal end of file size.</p> <p>Hello Olaf,<br/> Do you agree that this ticket can be closed as a dup of <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a>?<br/> Thanks again and in advance for your help and answer.</p> <p>Hi Bruno,<br/> Yes, I agree this should be closed as a dup of <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a>. Thank you.<br/> -Olaf</p> <p>Thanks Bruno and Olaf.<br/> ~ jfc.</p> Related LU-6556 LU-6634 Development Rank 1|hzxjr3: Rank (Obsolete) 9223372036854775807 Severity