https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6819 Lustre <p>After several MDT failovers (ldiskfs format) the MDT is failing to mount. Mount produces this LBUG:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jul 8 15:55:53 lola-11 kernel: LustreError: 38479:0:(ldlm_lib.c:2313:replay_request_or_update()) ASSERTION( tdtd-&gt;tdtd_last_update_transno &lt;= transno ) failed: Jul 8 15:55:53 lola-11 kernel: LustreError: 38479:0:(ldlm_lib.c:2313:replay_request_or_update()) LBUG </pre> </div></div> <p>Unfortunately panic_on_lbug was set. Will attempt to reproduce and get stack<br/> So far we have</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Call Trace:^M [&lt;ffffffffa0741875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M [&lt;ffffffffa0741e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs]^M [&lt;ffffffffa0a909e7&gt;] target_recovery_thread+0x2147/0x2720 [ptlrpc]^M [&lt;ffffffff8152a2be&gt;] ? thread_return+0x4e/0x7d0^M [&lt;ffffffffa0a8e8a0&gt;] ? target_recovery_thread+0x0/0x2720 [ptlrpc]^M [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0^M [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20^M [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0^M [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20^M </pre> </div></div> Lola - 2.7.56 tag LU-6819

LBUG ASSERTION( tdtd->tdtd_last_update_transno <= transno ) failed

Bug Blocker Resolved Fixed Di Wang Cliff White Wed, 8 Jul 2015 23:03:46 +0000 Wed, 26 Aug 2015 17:41:43 +0000 Wed, 26 Aug 2015 17:41:43 +0000 Lustre 2.8.0 Lustre 2.8.0 0 6 <p>Lustre log from repeat of LBUG:<br/> stack:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jul 8 16:28:59 lola-11 kernel: LustreError: 19026:0:(ldlm_lib.c:2313:replay_request_or_update()) ASSERTION( tdtd-&gt;tdtd_last_update_transno &lt;= transno ) failed: Jul 8 16:28:59 lola-11 kernel: LustreError: 19026:0:(ldlm_lib.c:2313:replay_request_or_update()) LBUG Jul 8 16:28:59 lola-11 kernel: Pid: 19026, comm: tgt_recov Jul 8 16:28:59 lola-11 kernel: Jul 8 16:28:59 lola-11 kernel: Call Trace: Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffffa0741875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffffa0741e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffffa0a909e7&gt;] target_recovery_thread+0x2147/0x2720 [ptlrpc] Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffff8152a2be&gt;] ? thread_return+0x4e/0x7d0 Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffffa0a8e8a0&gt;] ? target_recovery_thread+0x0/0x2720 [ptlrpc] Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 Jul 8 16:28:59 lola-11 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Jul 8 16:28:59 lola-11 kernel: </pre> </div></div> <p>According to the debug log, it seems update records in the replay list is out of order. But I can not any issue in the code.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 00000020:00080000:10.0:1436398138.407158:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8591034069 batchid = 4307872635 flags = 0 ops = 29 params = 21 00000020:00080000:10.0:1436398138.407160:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8591034071 batchid = 4307872636 flags = 0 ops = 10 params = 3 00000020:00080000:10.0:1436398138.407163:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8591034071 batchid = 4307872637 flags = 0 ops = 37 params = 9 00000020:00080000:10.0:1436398138.407165:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8591252290 batchid = 4307879249 flags = 0 ops = 67 params = 46 00000020:00080000:10.0:1436398138.407167:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8591252303 batchid = 4307879250 flags = 0 ops = 9 params = 5 00000020:00080000:10.0:1436398138.407169:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8590494596 batchid = 4307862210 flags = 0 ops = 4 params = 2 00000020:00080000:10.0:1436398138.407171:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8590494600 batchid = 4307862211 flags = 0 ops = 5 params = 8 00000020:00080000:10.0:1436398138.407173:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8590494602 batchid = 4307862212 flags = 0 ops = 4 params = 2 00000020:00080000:10.0:1436398138.407175:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8590494606 batchid = 4307862213 flags = 0 ops = 5 params = 8 00000020:00080000:10.0:1436398138.407177:0:19026:0:(update_records.c:72:update_records_dump()) master transno = 8590494608 batchid = 4307862214 flags = 0 ops = 4 params = 2 </pre> </div></div> <p>The list is supposed to be sorted by master transno, but seems not since 8590494596.</p> <p>Di, it looks like maybe the llog has wrapped around? What is the last_committed transno, and should those older ones be dropped?</p> <p>Andreas: Even the llog (on disk) has wrapped around. But after it reconstruct the llog record into update replay request and insert to the replay list, it will make sure the list is sorted and each item will be unique. I will investigate this further. Thanks. </p> <p>Cliff: I add some debug information in <a href="https://build.hpdd.intel.com/job/lustre-reviews/33316/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/33316/</a> . Could you please try this rpm and get me the debug log? Thanks</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15594" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15594</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6819" title="LBUG ASSERTION( tdtd-&gt;tdtd_last_update_transno &lt;= transno ) failed" class="issue-link" data-issue-key="LU-6819"><del>LU-6819</del></a> update: update transno of replay properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2e2b244de4b7094934eb3c9dd07d3d4af7f87fbd</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15594/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15594/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6819" title="LBUG ASSERTION( tdtd-&gt;tdtd_last_update_transno &lt;= transno ) failed" class="issue-link" data-issue-key="LU-6819"><del>LU-6819</del></a> update: update transno of replay properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: af13cfff297d4882de35fb7c11bf5261293b8287</p> <p>Landed for 2.8.</p> Related LU-6831 Development Rank 1|hzxhrr: Rank (Obsolete) 9223372036854775807 Severity