https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7490 Lustre <p>The error occurred during soak testing of master branch build '20151122' (see <a href="https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20151122" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20151122</a>). DNE is enabled. MDSes are configured in active-active failover configuration.</p> <p>Sequence of events:</p> <ul> <li>2015-11-26 10:32 Failover resources (mdt-0,1) <tt>lola-8</tt> --&gt; <tt>lola-9</tt> started</li> <li>2015-11-26 11:40 Failback resources (mdt-0.1) <tt>lola-9</tt> --&gt; <tt>lola-8</tt> completed successful</li> <li>2015-11-26 11:44 LBUG on lola-8. See the following message.</li> </ul> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 26 11:44:54 lola-8 kernel: LustreError: 8491:0:(out_lib.c:692:out_tx_write_exec()) read record [0x240089779:0x1:0x0] tail_pos 173122472 rc -53 index 50635 size 172659608 Nov 26 11:44:54 lola-8 kernel: LustreError: 8491:0:(out_lib.c:693:out_tx_write_exec()) LBUG Nov 26 11:44:54 lola-8 kernel: Pid: 8491, comm: mdt_out03_004 Nov 26 11:44:54 lola-8 kernel: Nov 26 11:44:54 lola-8 kernel: Call Trace: Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa07fb875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa07fbe77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa0bb60a0&gt;] out_tx_write_exec+0x500/0x7a0 [ptlrpc] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa0bb934b&gt;] ? out_tx_xattr_set_exec+0xeb/0x680 [ptlrpc] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa0bae13a&gt;] out_tx_end+0xda/0x5d0 [ptlrpc] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa0bb3726&gt;] out_handle+0xbd6/0x1890 [ptlrpc] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa0afa4e0&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa0baae1c&gt;] tgt_request_handle+0x8bc/0x12e0 [ptlrpc] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa0b52711&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffff8152a39e&gt;] ? thread_return+0x4e/0x7d0 Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffffa0b518d0&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 Nov 26 11:44:54 lola-8 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Nov 26 11:44:54 lola-8 kernel: Nov 26 11:44:54 lola-8 kernel: LustreError: dumping log to /tmp/lustre-log.1448567093.8491 </pre> </div></div> <p>Most likely this event is related to <a href="https://jira.whamcloud.com/browse/LU-7488" title="_req_capsule_get()) ASSERTION( fmt != ((void *)(long)0x5a5a5a5a5a5a5a5a) ) failed:" class="issue-link" data-issue-key="LU-7488"><del>LU-7488</del></a> which happened almost at the same time on the HA failover partner (<tt>lola-9</tt>)</p> <p>Attached console and messages log file of MDS (<tt>lola-8</tt>), kernel debug log file mentioned in the LBUG error message and error messages extracted from Lustre client nodes messages files that showed up at the same time.</p> lola<br/> build: 2.7.63-4-gf84e06e, a7eface85ea2d2aa6198681264b082a0244855d4 + patches LU-7490

out_tx_write_exec()) LBUG

Bug Blocker Resolved Fixed Di Wang Frank Heckes soak Fri, 27 Nov 2015 11:38:37 +0000 Thu, 28 Jan 2016 16:59:32 +0000 Thu, 28 Jan 2016 16:59:32 +0000 Lustre 2.8.0 Lustre 2.8.0 0 5 <p>According to the debug log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00080000:29.0:1448567093.939286:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939290:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:11.0:1448567093.939292:0:8215:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518918717705928) req@ffff88082ea09050 x1518918717714200/t0(0) o1000-&gt;soaked-MDT0003-mdtlov_UUID@192.168.1.109@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939297:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:11.0:1448567093.939298:0:8215:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518918717705928) req@ffff88082ea09050 x1518918717714200/t0(0) o1000-&gt;soaked-MDT0003-mdtlov_UUID@192.168.1.109@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939303:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:11.0:1448567093.939305:0:8215:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518918717705928) req@ffff88082ea09050 x1518918717714200/t0(0) o1000-&gt;soaked-MDT0003-mdtlov_UUID@192.168.1.109@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939310:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:11.0:1448567093.939316:0:8215:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518918717705928) req@ffff88082ea09050 x1518918717714200/t0(0) o1000-&gt;soaked-MDT0003-mdtlov_UUID@192.168.1.109@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939318:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939325:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:11.0:1448567093.939325:0:8215:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518918717705928) req@ffff88082ea09050 x1518918717714200/t0(0) o1000-&gt;soaked-MDT0003-mdtlov_UUID@192.168.1.109@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939332:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939336:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939340:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00080000:29.0:1448567093.939344:0:8491:0:(out_handler.c:89:out_check_resent()) @@@ no reply for RESENT req (have 1518924261645488) req@ffff8808040a5450 x1518924261650748/t0(0) o1000-&gt;soaked-MDT0006-mdtlov_UUID@192.168.1.111@o2ib10:694/0 lens 320/4320 e 0 to 0 dl 1448567099 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00020000:29.0:1448567093.940373:0:8491:0:(out_lib.c:692:out_tx_write_exec()) read record [0x240089779:0x1:0x0] tail_pos 173122472 rc -53 index 50635 size 172659608 00000020:00040000:29.0:1448567093.940379:0:8491:0:(out_lib.c:693:out_tx_write_exec()) LBUG </pre> </div></div> <p>It seems the request x1518924261650748 is not being invalidated for failure, which cause the corruption. sigh, there are no debug log on MDT0006, so I do not know what happens there.</p> <p>Hmm, it seems not all of resend requests are added to the delayed_list, </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* retry indefinitely on EINPROGRESS */ if (lustre_msg_get_status(req-&gt;rq_repmsg) == -EINPROGRESS &amp;&amp; ptlrpc_no_resend(req) == 0 &amp;&amp; !req-&gt;rq_no_retry_einprogress) { time_t now = cfs_time_current_sec(); DEBUG_REQ(D_RPCTRACE, req, "Resending request on EINPROGRESS"); spin_lock(&amp;req-&gt;rq_lock); req-&gt;rq_resend = 1; spin_unlock(&amp;req-&gt;rq_lock); req-&gt;rq_nr_resend++; /* Readjust the timeout for current conditions */ ptlrpc_at_set_req_timeout(req); /* delay resend to give a chance to the server to get ready. * The delay is increased by 1s on every resend and is capped to * the current request timeout (i.e. obd_timeout if AT is off, * or AT service time x 125% + 5s, see at_est2timeout) */ if (req-&gt;rq_nr_resend &gt; req-&gt;rq_timeout) req-&gt;rq_sent = now + req-&gt;rq_timeout; else req-&gt;rq_sent = now + req-&gt;rq_nr_resend; RETURN(0); } </pre> </div></div> <p>I will update <a href="http://review.whamcloud.com/#/c/17199/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17199/</a></p> <p>Here is the patch <a href="http://review.whamcloud.com/#/c/17199/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17199/</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17199/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17199/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7490" title="out_tx_write_exec()) LBUG" class="issue-link" data-issue-key="LU-7490"><del>LU-7490</del></a> recovery: abort update recovery once fails<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 92890d8f555d12ad32dc9841a328e84c5d26e896</p> <p>Landed for 2.8</p> Related LU-7488 LU-7488 LU-7455 Development End date Mon, 7 Dec 2015 11:38:37 +0000 Rank 1|hzxuf3: Rank (Obsolete) 9223372036854775807 Severity Start date Fri, 27 Nov 2015 11:38:37 +0000