https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5287 Lustre <p>Running racer with 2 clients MDSCOUNT=1 and 2.5.60-90-g37432a8 + <a href="http://review.whamcloud.com/#/c/5936/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5936/</a> I see this when restarting a crashed OST with some clients still mounted.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 230.089707] Lustre: Skipped 75 previous similar messages [ 231.775205] Lustre: 2151:0:(client.c:1924:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1404323793/real 1404323793] req@ffff8801f78fc110 x1472540086110788/t0(0) o400-&gt;lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 224/224 e 1 to 1 dl 1404323837 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1 [ 237.775938] Lustre: lustre-OST0001: Denying connection for new client cc64d6dc-4180-e700-9f7e-ce147524a8f0 (at 0@lo), waiting for all 4 known clients (2 recovered, 1 in progress, and 1 evicted) to recover in 0:36 [ 237.781858] Lustre: Skipped 3 previous similar messages [ 242.801254] LustreError: 2880:0:(ldlm_lib.c:2253:target_queue_recovery_request()) ASSERTION( req-&gt;rq_export-&gt;exp_lock_replay_needed ) failed: [ 242.805102] LustreError: 2880:0:(ldlm_lib.c:2253:target_queue_recovery_request()) LBUG [ 242.807953] Pid: 2880, comm: ll_ost00_007 [ 242.809274] [ 242.809276] Call Trace: [ 242.810585] [&lt;ffffffffa02b98c5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 242.812764] [&lt;ffffffffa02b9ec7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 242.814689] [&lt;ffffffffa064ea0c&gt;] target_queue_recovery_request+0xbac/0xc10 [ptlrpc] [ 242.816347] [&lt;ffffffffa06e122f&gt;] tgt_handle_recovery+0x38f/0x520 [ptlrpc] [ 242.817666] [&lt;ffffffffa06e6b8d&gt;] tgt_request_handle+0x18d/0xad0 [ptlrpc] [ 242.818987] [&lt;ffffffffa0699e31&gt;] ptlrpc_main+0xcf1/0x1880 [ptlrpc] [ 242.820261] [&lt;ffffffffa0699140&gt;] ? ptlrpc_main+0x0/0x1880 [ptlrpc] [ 242.821440] [&lt;ffffffff8109eab6&gt;] kthread+0x96/0xa0 [ 242.822360] [&lt;ffffffff8100c30a&gt;] child_rip+0xa/0x20 [ 242.823303] [&lt;ffffffff81554710&gt;] ? _spin_unlock_irq+0x30/0x40 [ 242.824390] [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [ 242.825391] [&lt;ffffffff8109ea20&gt;] ? kthread+0x0/0xa0 [ 242.826315] [&lt;ffffffff8100c300&gt;] ? child_rip+0x0/0x20 [ 242.827283] </pre> </div></div> LU-5287

(ldlm_lib.c:2253:target_queue_recovery_request()) ASSERTION( req->rq_export->exp_lock_replay_needed ) failed

Bug Critical Resolved Fixed Niu Yawei John Hammond llnl ost Wed, 2 Jul 2014 18:06:06 +0000 Mon, 19 Sep 2016 02:51:03 +0000 Thu, 6 Nov 2014 21:49:55 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.7.0 Lustre 2.5.4 0 11 <p>We hit this same assertion with Lustre version 2.4.2-14.1chaos (see github.com/chaos/lustre) while the OSTs were in recovery.</p> <p>Niu</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>replay-single/73c on review-dne-part-2 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/20dd03c0-365e-11e4-bd53-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/20dd03c0-365e-11e4-bd53-5254006e85c2</a></p> <p>The replay-single/73c seems never be really tested since the fail_loc OBD_FAIL_TGT_LAST_REPLAY was never be used (from the day one it was introduced)... Not sure if this test can trigger the bug more easily once it's fixed. I'm going to investigate it further.</p> <p>Well, I found that there are two places which modify the exp_flags without holding exp_lock, that could result in concurrent exp_flags updating overwrites each other.</p> <p>patch for master: <a href="http://review.whamcloud.com/11871" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11871</a></p> <p>Andriy discovered another path to trigger this assertion. (see <a href="https://jira.whamcloud.com/browse/LU-5651" title="ASSERTION( req-&gt;rq_export-&gt;exp_lock_replay_needed ) failed" class="issue-link" data-issue-key="LU-5651"><del>LU-5651</del></a>), patch is being reviewed on: <a href="http://review.whamcloud.com/#/c/12015/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12015/</a></p> <p>Hit this bug on master branch build #2671<br/> <a href="https://testing.hpdd.intel.com/test_sets/d986a3a2-472c-11e4-a9ec-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d986a3a2-472c-11e4-a9ec-5254006e85c2</a></p> <p>We just had 181 servers crash with this assertion when starting up 2.4.2-16chaos (see github.com/chaos/lustre) on the servers for the first time. We need a patch for our branch as well.</p> <p>Christopher, which patch(es) do you need back ported to b2_4? Don't want to do too much or too little.</p> <p>All of the patches needed to fix this ticket's assertion.</p> <p>But actually, we are moving to 2.5 soon, so patches for b2_5 should be sufficient.</p> <p>backports to b2_5:<br/> <a href="http://review.whamcloud.com/#/c/12162" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12162</a><br/> <a href="http://review.whamcloud.com/#/c/12163" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12163</a></p> <p>Landed for 2.5.4 and 2.7</p> <p>Hello Niu,</p> <p>I am trying to understand <a href="http://review.whamcloud.com/#/c/12162" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12162</a>.</p> <p>Can you please help me to understand how it affects <b>req-&gt;rq_export-&gt;exp_lock_replay_needed</b> flag ?</p> <p>Thanks in advance,</p> <p>All export flags share the same "unsigned long" and flag set operation "export-&gt;exp_xxx = 1" isn't atomic (load the whole unsigned long, change a bit, write the whole unsigned long), so flag change operation needs to take lock. </p> <p>Thanks Niu.</p> <p>Please correct me if wrong.</p> <blockquote><p>so flag change operation needs to take lock</p></blockquote> <p>If the operations are not concurrent then it does not matter whether bit filed change happens with lock or without lock. The whole unsigned long will not be affected if we change any of the bits. Am I right ?</p> <blockquote><p>load the whole unsigned long, change a bit, write the whole unsigned long</p></blockquote> <p>what is the real importance of lock here ?</p> <p>Thanks in advance,</p> <p>Right, it's safe if there isn't any concurrent changes.</p> <p>The importance of lock here is that there could be concurrent changes, I didn't take a thoroughly retrospect on the code changes, it looks concurrency is possible at first glance. Even if no concurrent changes in present code, I don't think we can hypothesize that concurrent changes will never happen, taking lock is the safe way to avoid nasty bugs.</p> <p>Thanks Niu.</p> <p>We have faced similar issue on our set up. I will update my further investigation here.</p> Duplicate LU-5572 Related LU-5651 Development Rank 1|hzwqg7: Rank (Obsolete) 14750 Severity