https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2397 Lustre <p>I did a quick search of the existing bugs, but didn't find this one.</p> <p>Running the <tt>replay-single</tt> test in a single node VM setup I triggered the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LustreError: 14716:0:(ldlm_lib.c:1749:check_for_next_transno()) ASSERTION( req_transno &gt;= next_transno ) failed: req_transno: 0, next_transno: 51539608157 LustreError: 14716:0:(ldlm_lib.c:1749:check_for_next_transno()) LBUG Kernel panic - not syncing: LBUG Pid: 14716, comm: tgt_recov Tainted: P --------------- 2.6.32-279.9.1.1chaos.ch5.1.x86_64 #1 Call Trace: [&lt;ffffffff814fdceb&gt;] ? panic+0xa0/0x168 [&lt;ffffffffa0705f6b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0991e75&gt;] ? check_for_next_transno+0x585/0x590 [ptlrpc] [&lt;ffffffffa09918f0&gt;] ? check_for_next_transno+0x0/0x590 [ptlrpc] [&lt;ffffffffa098be0e&gt;] ? target_recovery_overseer+0x5e/0x250 [ptlrpc] [&lt;ffffffffa098a270&gt;] ? exp_req_replay_healthy+0x0/0x30 [ptlrpc] [&lt;ffffffffa0716591&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0992f55&gt;] ? target_recovery_thread+0x7b5/0x19d0 [ptlrpc] [&lt;ffffffffa09927a0&gt;] ? target_recovery_thread+0x0/0x19d0 [ptlrpc] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffffa09927a0&gt;] ? target_recovery_thread+0x0/0x19d0 [ptlrpc] [&lt;ffffffffa09927a0&gt;] ? target_recovery_thread+0x0/0x19d0 [ptlrpc] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>The test case was simply:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ sudo ONLY="59 60" FSTYPE=zfs ./lustre/tests/replay-single.sh </pre> </div></div> LU-2397

Assertion triggered in check_for_next_transno

Bug Blocker Resolved Fixed Mikhail Pershin Prakash Surya LB Tue, 27 Nov 2012 17:11:45 +0000 Fri, 19 Apr 2013 22:30:41 +0000 Fri, 18 Jan 2013 14:20:52 +0000 Lustre 2.4.0 Lustre 2.4.0 0 6 <p>Could you please comment on this one? Thanks Peter</p> <p>replay with 0 transno? Looks wrong. Is that reproduced easily with command you've mentioned?</p> <p>I only hit it once while running that test in a loop for about 6 hours.. I wouldn't call that easy to reproduce. Honestly, there are probably more pressing issues to spend time on, but I just wanted to document this here since I did hit it once.</p> <p>I hit this too, in recovery-ost-single, test 2</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[107343.839932] Lustre: 26202:0:(ldlm_lib.c:2184:target_recovery_init()) RECOVERY: service lustre-OST0000, 2 recoverable clients, last_transno 12884901891 [107348.833481] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [107348.836301] LustreError: 26224:0:(ldlm_lib.c:1748:check_for_next_transno()) ASSERTION( req_transno &gt;= next_transno ) failed: req_transno: 0, next_transno: 12884901892 [107348.839179] LustreError: 26224:0:(ldlm_lib.c:1748:check_for_next_transno()) LBUG [107348.840790] Pid: 26224, comm: tgt_recov [107348.841717] [107348.841720] Call Trace: [107348.842588] [&lt;ffffffffa0ae8915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [107348.842866] [&lt;ffffffffa0ae8f27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [107348.843157] [&lt;ffffffffa1331b36&gt;] check_for_next_transno+0x596/0x5a0 [ptlrpc] [107348.843598] [&lt;ffffffffa13315a0&gt;] ? check_for_next_transno+0x0/0x5a0 [ptlrpc] [107348.844039] [&lt;ffffffffa132bb46&gt;] target_recovery_overseer+0x66/0x230 [ptlrpc] [107348.844532] [&lt;ffffffffa1329fb0&gt;] ? exp_req_replay_healthy+0x0/0x30 [ptlrpc] [107348.844827] [&lt;ffffffffa0af9641&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [107348.845169] [&lt;ffffffffa1332ae3&gt;] target_recovery_thread+0x683/0x1660 [ptlrpc] [107348.845686] [&lt;ffffffff814faef5&gt;] ? _spin_unlock_irq+0x15/0x20 [107348.846050] [&lt;ffffffffa1332460&gt;] ? target_recovery_thread+0x0/0x1660 [ptlrpc] [107348.846522] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [107348.846795] [&lt;ffffffffa1332460&gt;] ? target_recovery_thread+0x0/0x1660 [ptlrpc] [107348.847313] [&lt;ffffffffa1332460&gt;] ? target_recovery_thread+0x0/0x1660 [ptlrpc] [107348.847798] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [107348.848056] [107348.848285] BUG: spinlock cpu recursion on CPU#3, ll_ost01_003/25839 (Not tainted) [107348.848750] lock: ffff88004d00cc38, .magic: dead4ead, .owner: tgt_recov/26224, .owner_cpu: 3 [107348.849257] Pid: 25839, comm: ll_ost01_003 Not tainted 2.6.32-debug #6 [107348.849558] Call Trace: [107348.849764] [&lt;ffffffff8128098a&gt;] ? spin_bug+0xaa/0x100 [107348.850006] [&lt;ffffffff81280ba1&gt;] ? _raw_spin_lock+0x121/0x180 [107348.850290] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [107348.850588] [&lt;ffffffff814fafde&gt;] ? _spin_lock+0xe/0x10 [107348.850872] [&lt;ffffffffa1330d6c&gt;] ? target_queue_recovery_request+0x41c/0xc50 [ptlrpc] [107348.851438] [&lt;ffffffff814faf3e&gt;] ? _spin_unlock+0xe/0x10 [107348.851617] Kernel panic - not syncing: LBUG [107348.851621] Pid: 26224, comm: tgt_recov Not tainted 2.6.32-debug #6 [107348.851622] Call Trace: [107348.851629] [&lt;ffffffff814f75e4&gt;] ? panic+0xa0/0x168 [107348.851642] [&lt;ffffffffa0ae8f7b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [107348.851673] [&lt;ffffffffa1331b36&gt;] ? check_for_next_transno+0x596/0x5a0 [ptlrpc] [107348.851701] [&lt;ffffffffa13315a0&gt;] ? check_for_next_transno+0x0/0x5a0 [ptlrpc] [107348.851728] [&lt;ffffffffa132bb46&gt;] ? target_recovery_overseer+0x66/0x230 [ptlrpc] [107348.851755] [&lt;ffffffffa1329fb0&gt;] ? exp_req_replay_healthy+0x0/0x30 [ptlrpc] [107348.851768] [&lt;ffffffffa0af9641&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [107348.851796] [&lt;ffffffffa1332ae3&gt;] ? target_recovery_thread+0x683/0x1660 [ptlrpc] [107348.851799] [&lt;ffffffff814faef5&gt;] ? _spin_unlock_irq+0x15/0x20 [107348.851826] [&lt;ffffffffa1332460&gt;] ? target_recovery_thread+0x0/0x1660 [ptlrpc] [107348.851829] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [107348.851856] [&lt;ffffffffa1332460&gt;] ? target_recovery_thread+0x0/0x1660 [ptlrpc] [107348.851883] [&lt;ffffffffa1332460&gt;] ? target_recovery_thread+0x0/0x1660 [ptlrpc] [107348.851886] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>crashdump and modules are in /exports/crashdumps/192.168.10.219-2012-12-30-10\:28\:47</p> <p>The code below is not safe:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (cfs_atomic_read(&amp;obd-&gt;obd_req_replay_clients) == 0) { CDEBUG(D_HA, <span class="code-quote">"waking <span class="code-keyword">for</span> completed recovery\n"</span>); wake_up = 1; } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (req_transno == next_transno) { CDEBUG(D_HA, <span class="code-quote">"waking <span class="code-keyword">for</span> next ("</span>LPD64<span class="code-quote">")\n"</span>, next_transno); wake_up = 1; } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (queue_len == cfs_atomic_read(&amp;obd-&gt;obd_req_replay_clients)) { <span class="code-object">int</span> d_lvl = D_HA; <span class="code-comment">/** handle gaps occured due to lost reply or VBR */</span> LASSERTF(req_transno &gt;= next_transno, <span class="code-quote">"req_transno: "</span>LPU64<span class="code-quote">", next_transno: "</span>LPU64<span class="code-quote">"\n"</span>, req_transno, next_transno); </pre> </div></div> <p>the obd_req_replay_clients counter is checked for 0 and after that it is supposed to be not zero in next check against queue_len. I believe that was so in past when all counters were changed under the same lock but some time ago they became atomic values and many places became lockless due to smp improvements work. So now we can't rely on the fact that this counter can not change. Those recent issues described in this ticket are about that - queue_len is 0 and counter become 0 exactly at second check, so we enter the path which is not supposed to be run with zero queue_len.</p> <p>The fix would be check queue_len &gt; 0 in addition to the existing check, also obd_req_replay_clients check for 0 should be done after comparision with queue_len.</p> <p><a href="http://review.whamcloud.com/#change,4998" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4998</a></p> <p>landed</p> Development Rank 1|hzvcxj: Rank (Obsolete) 5688 Severity