https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3274 Lustre <p>Hit this running replay-single in a loop</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[123247.989106] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) == 00:41:34 (1367728894) [123248.537768] Turning device loop1 (0x700001) read-only [123248.562834] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [123248.580423] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [123250.719913] Lustre: DEBUG MARKER: cancel_lru_locks osc start [123250.880677] Lustre: DEBUG MARKER: cancel_lru_locks osc stop [123251.389028] Removing read-only on unknown block (0x700001) [123263.314546] LDISKFS-fs (loop1): recovery complete [123263.357806] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=on. Opts: [123268.399242] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-0@lo inode [0x2000061c0:0x5:0x0] object 0x0:7458 extent [0-1048575]: client csum 7945acf6, server csum b9e6e441 [123268.477228] LustreError: 17404:0:(osc_cache.c:1774:osc_dec_unstable_pages()) ASSERTION( atomic_read(&amp;cli-&gt;cl_cache-&gt;ccc_unstable_nr) &gt;= 0 ) failed: [123268.477757] LustreError: 17404:0:(osc_cache.c:1774:osc_dec_unstable_pages()) LBUG [123268.478168] Pid: 17404, comm: ptlrpcd_rcv [123268.478381] [123268.478381] Call Trace: [123268.478752] [&lt;ffffffffa0e018a5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [123268.479019] [&lt;ffffffffa0e01ea7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [123268.479981] [&lt;ffffffffa0458bcc&gt;] osc_dec_unstable_pages+0x12c/0x190 [osc] [123268.480270] [&lt;ffffffffa114d76b&gt;] ptlrpc_free_committed+0x14b/0x620 [ptlrpc] [123268.480593] [&lt;ffffffffa114f4e3&gt;] after_reply+0x7a3/0xd90 [ptlrpc] [123268.480871] [&lt;ffffffffa1154493&gt;] ptlrpc_check_set+0x1093/0x1da0 [ptlrpc] [123268.481150] [&lt;ffffffffa1180e2b&gt;] ptlrpcd_check+0x55b/0x590 [ptlrpc] [123268.481420] [&lt;ffffffffa1181373&gt;] ptlrpcd+0x233/0x390 [ptlrpc] [123268.481665] [&lt;ffffffff8105ad10&gt;] ? default_wake_function+0x0/0x20 [123268.481963] [&lt;ffffffffa1181140&gt;] ? ptlrpcd+0x0/0x390 [ptlrpc] [123268.482220] [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 [123268.482465] [&lt;ffffffffa1181140&gt;] ? ptlrpcd+0x0/0x390 [ptlrpc] [123268.482733] [&lt;ffffffffa1181140&gt;] ? ptlrpcd+0x0/0x390 [ptlrpc] [123268.482978] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 [123268.483209] [123268.543751] Kernel panic - not syncing: LBUG </pre> </div></div> <p>This is somewhat current master plust 3 more lu-2139 patches applied on top.<br/> I have a crashdump in /exports/crashdumps/192.168.10.221-2013-05-05-00\:41\:57/<br/> code tag: master-20130505</p> LU-3274

osc_cache.c:1774:osc_dec_unstable_pages()) ASSERTION( atomic_read(&cli->cl_cache->ccc_unstable_nr) >= 0 ) failed

Bug Major Resolved Fixed Jinshan Xiong Oleg Drokin Sun, 5 May 2013 04:48:53 +0000 Fri, 7 Nov 2014 08:30:44 +0000 Wed, 19 Mar 2014 17:08:51 +0000 Lustre 2.4.0 Lustre 2.6.0 0 8 <p>Got another hit, replay-single again /exports/crashdumps/192.168.10.216-2013-05-06-02\:27\:44/vmcore<br/> This time in replay-single test 87, apparently there's some mismatch on replay? Double accounting by any chance?</p> <p>Hmm, I never saw this failure during my testing. The asssertion was added because the unstable count for a given FS should never become negative, which looks to be happening here. I'm curious if the commit callback is getting called twice for certain requests, once directly <tt>after_reply</tt>, and then again in <tt>ptlrpc_free_committed</tt> (via <tt>after_reply</tt>). I'll read through the patch again and see if I can piece together what might be happening here. Was this failure triggered through Maloo?</p> <p>no, it was not triggered on maloo, it's on my private burn-in cluster that promotes race conditions. But if you are interested, i can make the vmcore, modules and the code snapshot available somehow.</p> <p>OK. Let me see if I can get a possible explanation out of reading the code first, and I might want to look at those later to back up my theory.</p> <p>I looked over the code some more and I'm still unsure how this would occur. I don't think it is a result of <tt>rq_commit_cb</tt> getting called twice for the same request (via <tt>after_reply</tt> and <tt>ptlrpc_free_committed</tt>). AFAICT it will get run directly in <tt>after_reply</tt>, or get added to the <tt>imp_replay_list</tt> and run later via <tt>ptlrpc_free_committed</tt>, but not both. I'm running replay-single in a loop in a single node VM setup, but haven't hit the bug yet. Oleg, if you have some spare cycles it <em>might</em> be interesting to add this patch:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> diff --git i/lustre/osc/osc_request.c w/lustre/osc/osc_request.c index a1c740c..ac2eaf5 100644 --- i/lustre/osc/osc_request.c +++ w/lustre/osc/osc_request.c @@ -2165,6 +2165,8 @@ int osc_build_rpc(const struct lu_env *env, struct client_obd *cli, GOTO(out, rc); } + LASSERT(req-&gt;rq_committed == 0); + LASSERT(req-&gt;rq_unstable == 0); req-&gt;rq_commit_cb = brw_commit; req-&gt;rq_interpret_reply = brw_interpret; </pre> </div></div> <p>I assumed these values would be already initialized to 0, but perhaps I'm wrong. If we allocate a request and it <em>happens</em> to have <tt>rq_unstable</tt> set to 1, and then <tt>brw_commit</tt> is called via <tt>rq_commit_cb</tt> without <tt>osc_inc_unstable_pages</tt> first being called, I could see this failure happening. But I have no evidence of this yet.</p> <p>it occurred in current master at: <a href="https://maloo.whamcloud.com/sub_tests/6f5c1b80-47cb-11e3-a71f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/6f5c1b80-47cb-11e3-a71f-52540035b04c</a> in the console message of client 2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>23:26:13:LustreError: 25093:0:(osc_cache.c:1807:osc_dec_unstable_pages()) ASSERTION( atomic_read(&amp;cli-&gt;cl_cache-&gt;ccc_unstable_nr) &gt;= 0 ) failed: 23:26:13:LustreError: 25093:0:(osc_cache.c:1807:osc_dec_unstable_pages()) LBUG 23:26:13:Pid: 25093, comm: ptlrpcd_rcv 23:26:13: 23:26:13:Call Trace: 23:26:13: [&lt;ffffffffa0b58895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 23:26:14: [&lt;ffffffffa0b58e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 23:26:14: [&lt;ffffffffa0f64f33&gt;] osc_dec_unstable_pages+0x133/0x1a0 [osc] 23:26:14: [&lt;ffffffffa0d601c9&gt;] ptlrpc_free_committed+0x149/0x620 [ptlrpc] 23:26:14: [&lt;ffffffffa0d61a74&gt;] after_reply+0x7a4/0xd90 [ptlrpc] 23:26:14: [&lt;ffffffffa0d66ab1&gt;] ptlrpc_check_set+0xf71/0x1b40 [ptlrpc] 23:26:15: [&lt;ffffffffa0d9120b&gt;] ptlrpcd_check+0x53b/0x560 [ptlrpc] 23:26:15: [&lt;ffffffffa0d9185b&gt;] ptlrpcd+0x33b/0x3f0 [ptlrpc] 23:26:15: [&lt;ffffffff81063990&gt;] ? default_wake_function+0x0/0x20 23:26:15: [&lt;ffffffffa0d91520&gt;] ? ptlrpcd+0x0/0x3f0 [ptlrpc] 23:26:15: [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0 23:26:15: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 23:26:15: [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 23:26:16: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 23:26:16: 23:26:16:Kernel panic - not syncing: LBUG 23:26:16:Pid: 25093, comm: ptlrpcd_rcv Not tainted 2.6.32-358.23.2.el6.x86_64 #1 23:26:16:Call Trace: 23:26:16: [&lt;ffffffff8150daac&gt;] ? panic+0xa7/0x16f 23:26:17: [&lt;ffffffffa0b58eeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 23:26:17: [&lt;ffffffffa0f64f33&gt;] ? osc_dec_unstable_pages+0x133/0x1a0 [osc] 23:26:17: [&lt;ffffffffa0d601c9&gt;] ? ptlrpc_free_committed+0x149/0x620 [ptlrpc] 23:26:17: [&lt;ffffffffa0d61a74&gt;] ? after_reply+0x7a4/0xd90 [ptlrpc] 23:26:17: [&lt;ffffffffa0d66ab1&gt;] ? ptlrpc_check_set+0xf71/0x1b40 [ptlrpc] 23:26:17: [&lt;ffffffffa0d9120b&gt;] ? ptlrpcd_check+0x53b/0x560 [ptlrpc] 23:26:17: [&lt;ffffffffa0d9185b&gt;] ? ptlrpcd+0x33b/0x3f0 [ptlrpc] 23:26:18: [&lt;ffffffff81063990&gt;] ? default_wake_function+0x0/0x20 23:26:18: [&lt;ffffffffa0d91520&gt;] ? ptlrpcd+0x0/0x3f0 [ptlrpc] 23:26:18: [&lt;ffffffff81096a36&gt;] ? kthread+0x96/0xa0 23:26:18: [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 23:26:19: [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 23:26:19: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>patch is at: <a href="http://review.whamcloud.com/8215" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8215</a></p> <p>Jinshan, please see <a href="http://review.whamcloud.com/8219" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8219</a> . I tried to make a more understandable version of your fix from 8215. What do you think?</p> <p>Patch landed to Master. Please reopen ticket if more work is needed.</p> <p>Here is the back-ported patch for Lustre b2_5 branch: <a href="http://review.whamcloud.com/12613" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12613</a><br/> The patch depends on the patches for <a href="https://jira.whamcloud.com/browse/LU-2139" title="Tracking unstable pages" class="issue-link" data-issue-key="LU-2139"><del>LU-2139</del></a> on Lustre b2_5 branch.</p> Blocker LU-2139 Development Rank 1|hzvq5r: Rank (Obsolete) 8111 Severity