https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7558 Lustre <p>/bgsys/logs/BGQ.sn/R04-ID-J00.log (among many others)</p> <p>LustreError: 28558:0: (niobuf.c:721:ptl_send_rpc()) ASSERTION( (at_max == 0) || request-&gt;rq_import-&gt;imp_state != LUSTRE_IMP_FULL || (request-&gt;rq_import-&gt;imp_msghdr_flags &amp; 0x1) || ! (request-&gt;rq_import-&gt;imp_connect_data.ocd_connect_flags &amp; 0x1000000ULL) ) failed:<br/> LustreError: 28558:0: (niobuf.c:721:ptl_send_rpc()) LBUG<br/> Call Trace:<br/> show_stack<br/> libcfs_debug_dumpstack<br/> lbug_with_loc<br/> ptl_send_rpc<br/> ptlrpc_send_new_req<br/> ptlrpc_set_wait<br/> ll_statfs_internal<br/> ll_statfs<br/> statfs_by_dentry<br/> vfs_statfs<br/> user_statfs<br/> SyS_statfs<br/> syscall_exit</p> <p>Occurred on many tens of I/O nodes, then within the next 24 hours, occurred on many tens more. Continuing to occur.</p> <p>We have not seen this issue before. The patch that introduced this assert was in the patch stack for our tag 2.5.4-1chaos, rolled out in April. We do not know what triggered this now.</p> <p>c389652 <a href="https://jira.whamcloud.com/browse/LU-5528" title="Race - connect vs resend" class="issue-link" data-issue-key="LU-5528"><del>LU-5528</del></a> ptlrpc: fix race between connect vs resend</p> <p>There are no crash dumps for these nodes, nor much in the console logs.</p> <p>Because several conditions were ASSERTed in a single statement, which failed is unknown.</p> BG/Q I/O nodes<br/> lustre-client-ion-2.5.4-4chaos_2.6.32_504.8.2.bgq.3blueos.V1R2M3.bl2.2_1.ppc64.ppc64<br/> LU-7558

niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags & MSGHDR_AT_SUPPORT ...)

Bug Critical Resolved Fixed Mikhail Pershin Olaf Faaland llnl Tue, 15 Dec 2015 17:48:32 +0000 Mon, 8 Apr 2019 14:07:59 +0000 Wed, 16 Jan 2019 13:06:15 +0000 Lustre 2.12.0 Lustre 2.9.0 Lustre 2.13.0 Lustre 2.10.7 Lustre 2.12.1 0 14 <p>Patch stack running on those nodes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">* ae35897 (tag: 2.5.4-4chaos) LU-2232 debug: print debug <span class="code-keyword">for</span> prolonged lock * 458c1f9 Revert <span class="code-quote">"LU-2232 debug: lock handle in IO request may be stale"</span> * f69e69e (tag: 2.5.4-3chaos) LU-6389 llite: restart <span class="code-object">short</span> read/write <span class="code-keyword">for</span> normal IO * 1e8f823 Revert <span class="code-quote">"LU-6389 llite: restart <span class="code-object">short</span> read/write <span class="code-keyword">for</span> normal IO"</span> * 3028245 (tag: 2.5.4-2chaos) LU-6152 osd-zfs: ZFS large block compat * fb6b94c Revert <span class="code-quote">"LU-5053 ptlrpc: Add schedule point to ptlrpc_check_set()"</span> * 0dc7041 Fix ldiskfs source autodetect <span class="code-keyword">for</span> CentOS 6 * 42edddb LU-6536 llapi: lmm_stripe_count used unswabbed * 3d25e65 LU-2182 llapi: implementation of <span class="code-keyword">new</span> llapi_layout API * fe5c3dd LLNL-0000 llapi: get OST count from proc * c1c8672 LU-4107 build: fix lustre_user.h to C++ compatible * f139e17 LU-5042 ldlm: delay filling resource's LVB upon replay * 35e5a78 (tag: 2.5.4-1chaos) LU-6389 llite: restart <span class="code-object">short</span> read/write <span class="code-keyword">for</span> normal IO * a25d1d1 LU-1032 build: Honor --disable-modules option in spec file * 01df569 LU-1032 build: Add Lustre DKMS spec file * 1f47ba6 LU-6038 osd-zfs: sa_spill_alloc()/sa_spill_free() compat * 16b79c6 LU-6038 osd-zfs: Avoid redefining KM_SLEEP * 36eca32 LU-5326 libcfs: remove umode_t typedef * 685c9ac LU-3353 ptlrpc: Suppress error message when imp_sec is freed * 9ea4c83 LU-5984 obd: fix lastid file name in compat code * c389652 LU-5528 ptlrpc: fix race between connect vs resend * bba812b LU-5579 ldlm: re-sent enqueue vs lock destroy race </pre> </div></div> <p>The console logs indicate a pair of OSS's (failover pair, in fact) had crashed and come back up. This error occured when they had timed out of recovery and evicted the clients. There's nothing else in the console log. The nodes are unresponsive after they hit this LBUG, so I'm not able to gather more information unless I can reproduce the situation while monitoring the node. Not clear right now how to do that without affecting the whole machine.</p> <p>Seems wrong to ASSERT anything to do with connection state when not holding a lock on the import. Also not helpful to ASSERT several conditions collectively instead of individually so the bad state is clearer. So I'm looking into changing to code to return failure instead.</p> <p>My statement about ASSERTing collectively is mistaken - OR not AND. But this still doesn't produce enough useful information, the patch will need to change that.</p> <p>I don't understand why they chose to ASSERT here instead of fail. There is discussion of this ASSERT in code review from the patch, but I don't see a rationale for crashing the node.</p> <p>Mike</p> <p>Could you please advise?</p> <p>Peter</p> <p>This bug is taking out a large number of the I/O Nodes on Sequoia, our largest production supercomputer, so I bumped the severity to 1.</p> <p>Chris</p> <p>Just to acknowledge this - I am discussing with engineering</p> <p>Peter</p> <p>Is 2.5.4-15chaos available somewhere? I do not see it on <a href="https://github.com/chaos/lustre" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre</a>.</p> <p>John</p> <p>I can direct you to it</p> <p>Peter</p> <p>Olaf</p> <p>When did the last software update get applied to these nodes and which tag preceded it? Have any changes to the configuration been made in recent weeks?</p> <p>Peter</p> <p>The nodes are only running 2.5.4-4chaos, the newer tags haven't made it to the BG/Q systems.</p> <p>No software changes have happened on these BG/Q clients in months. No recent change on the servers either.</p> <p>Servers are at lustre-2.5.4-13chaos</p> <p>The possible case (but I wan't sure about it!) could cause this problem is the race between setting the state of the obd_import</p> <p>in "ptlrpc_connect_import", the "imp_state" is set as "LUSTRE_IMP_CONNECTING", and if the check of the "imp_state"<br/> in "ptlrpc_import_recovery_state_machine" has been passed before this setting in "ptlrpc_connect_import", the "imp_state"<br/> could be set as "LUSTRE_IMP_FULL" by the delayed reply of last ping of recovery in "signal_completed_replay".</p> <p>the race is,<br/> thread 1: ptlrpc_import_recovery_state_machine<br/> ...<br/> if (imp-&gt;imp_state == LUSTRE_IMP_RECOVER) {<br/> struct ptlrpc_connection *conn = imp-&gt;imp_connection;</p> <p> rc = ptlrpc_resend(imp);<br/> if (rc)<br/> GOTO(out, rc);</p> <p> IMPORT_SET_STATE(imp, LUSTRE_IMP_FULL);<br/> ptlrpc_activate_import(imp);<br/> ...<br/> thread2: ptlrpc_connect_import is called during calling ptlrpc_resend, and the imp_state is set LUSTRE_IMP_CONNECTING and<br/> MSGHDR_AT_SUPPORT is cleared.</p> <p>thread1: the imp_state is set LUSTRE_IMP_FULL, and the LBUG is triggered.</p> <p>this assertion was added by patch for <a href="https://jira.whamcloud.com/browse/LU-5528" title="Race - connect vs resend" class="issue-link" data-issue-key="LU-5528"><del>LU-5528</del></a> ( c389652 <a href="https://jira.whamcloud.com/browse/LU-5528" title="Race - connect vs resend" class="issue-link" data-issue-key="LU-5528"><del>LU-5528</del></a> ptlrpc: fix race between connect vs resend ) so this situation is result of that commit. So either that patch is incomplete or assertion is not correct.</p> <p>What is special about LUSTRE_IMP_FULL? Why would the assertion check for that state, but none of the other import states like _EVICTED or _CLOSED?</p> <p>Also, there are many locations in the code within ptl_send_rpc() where something can cause the function to return an error. Do you know why the person who wrote the patch chose to ASSERT instead of return an error on import state?</p> <p>thanks</p> <p>Olaf, it looks for me like assertion was added to prevent similar situations as <a href="https://jira.whamcloud.com/browse/LU-5528" title="Race - connect vs resend" class="issue-link" data-issue-key="LU-5528"><del>LU-5528</del></a> fixed, but I agree that error handling with a message about details is better. Is it possible to revert this patch for now? We are going to check the correctness of that assertion and replace it with error handling instead.</p> <p>Mikhail, yes, we could revert the patch. I need to talk it over with Chris and the BG team. It's trickier than it normally would be, because lots of people are out of the office next week and so it's difficult to handle anything new that comes up.</p> <p>Should we add any debug code?</p> <p>Olaf, let me think a bit, I will provide patch with debug instead of assertion</p> <p>I agree with Hongchao, this assertion is result of race he mentioned, and that race itself was introduced by <a href="https://jira.whamcloud.com/browse/LU-5528" title="Race - connect vs resend" class="issue-link" data-issue-key="LU-5528"><del>LU-5528</del></a> patch. It fixes the case when import flags are set after import state is set to LUSTRE_IMP_FULL but open new race when flags are set before LUSTRE_IMP_FULL. Strictly speaking, both import flags and its state should be changed together.</p> <p>Olaf, I think it is better to revert this commit, even if we will remove that assertion, the import will still have wrong flags causing all requests to behave wrongly, so this situation is not better than before that commit. Meanwhile I will think how to solve this properly, I don't see quick solution so far.</p> <p>The possible solution I have in mind is to set new import bool value imp_connected which would indicate that import was reconnected (and imp flags are updated already) even if it is not in FULL state yet. So ptlrpc_connect_import() will not try to initiate new connect by checking this flag (now it checks import state) and will not clear flags for import which is connected but not FULL yet. I am not sure this is the only solution, it has to be discussed.</p> <p>Mikhail, OK, we are reverting this patch for now.</p> <p>This is still broken on master and b2_8. Since we hope to move to 2.8 in the near future, we would really like to see this fixed as soon as possible.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19312" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19312</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7558" title="niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)" class="issue-link" data-issue-key="LU-7558"><del>LU-7558</del></a> import: don't reconnect during connect interpret<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 02f60229e4abb25e7da20ebaebe7e7bca18e99c3</p> <p>the patch implements approach I mentioned in comment above. It blocks new connect requests while current connect interpret is running, preventing the import connection flags re-setting.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19312/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19312/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7558" title="niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)" class="issue-link" data-issue-key="LU-7558"><del>LU-7558</del></a> import: don't reconnect during connect interpret<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d10320bafdb942a8dbc5a8ba9176873134a5ffa3</p> <p>Landed to master for 2.9.0</p> <p>Mikhail,</p> <p>i think it not a very good approach. It's easy to use a simple change like <br/> if (imp-&gt;imp_state =! LUSTRE_IMP_DISCONN ) instead of add a new flag.</p> <p>I am still hitting this regularly in my testing.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[16176.517655] LustreError: 2151:0:(client.c:1179:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8802f4d5fc80 x1608074126206960/t0(0) o38-&gt;lustre-MDT0000-mdc-ffff8802acac9800@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 [16178.021121] LustreError: 2309:0:(niobuf.c:782:ptl_send_rpc()) ASSERTION( (at_max == 0) || imp-&gt;imp_state != LUSTRE_IMP_FULL || (imp-&gt;imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT) || !(imp-&gt;imp_connect_data.ocd_connect_flags &amp; 0x1000000ULL) ) failed: [16178.032291] LustreError: 2309:0:(niobuf.c:782:ptl_send_rpc()) LBUG [16178.033617] CPU: 1 PID: 2309 Comm: rm Kdump: loaded Tainted: P OE ------------ 3.10.0-7.5-debug #1 [16178.036021] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [16178.037385] Call Trace: [16178.038659] [&lt;ffffffff8176fc9a&gt;] dump_stack+0x19/0x1b [16178.042848] [&lt;ffffffffa01b37c2&gt;] libcfs_call_trace+0x72/0x80 [libcfs] [16178.044440] [&lt;ffffffffa01b384c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [16178.046617] [&lt;ffffffffa05854e9&gt;] ptl_send_rpc+0xb79/0xe80 [ptlrpc] [16178.047953] [&lt;ffffffffa01b9f97&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [16178.049215] [&lt;ffffffffa0579d40&gt;] ptlrpc_send_new_req+0x460/0xa70 [ptlrpc] [16178.050257] [&lt;ffffffffa057e819&gt;] ptlrpc_set_wait+0x289/0x790 [ptlrpc] [16178.051188] [&lt;ffffffff810630a3&gt;] ? kvm_clock_read+0x33/0x40 [16178.052007] [&lt;ffffffff810630b9&gt;] ? kvm_clock_get_cycles+0x9/0x10 [16178.053405] [&lt;ffffffffa058abda&gt;] ? lustre_msg_set_jobid+0x9a/0x110 [ptlrpc] [16178.054486] [&lt;ffffffffa057ed9d&gt;] ptlrpc_queue_wait+0x7d/0x220 [ptlrpc] [16178.055636] [&lt;ffffffffa08a3ca7&gt;] mdc_reint+0x57/0x160 [mdc] [16178.056599] [&lt;ffffffffa08a4df6&gt;] mdc_unlink+0x206/0x450 [mdc] [16178.057700] [&lt;ffffffffa0500125&gt;] lmv_unlink+0x5a5/0x950 [lmv] [16178.059013] [&lt;ffffffffa146c764&gt;] ? ll_i2gids+0x24/0xb0 [lustre] [16178.060357] [&lt;ffffffffa1471b31&gt;] ll_unlink+0x171/0x5e0 [lustre] [16178.061621] [&lt;ffffffff8121d326&gt;] vfs_unlink+0x106/0x190 [16178.062807] [&lt;ffffffff8121ff6e&gt;] do_unlinkat+0x26e/0x2b0 [16178.063937] [&lt;ffffffff8178387b&gt;] ? system_call_after_swapgs+0xc8/0x160 [16178.065792] [&lt;ffffffff8178386f&gt;] ? system_call_after_swapgs+0xbc/0x160 [16178.068348] [&lt;ffffffff8178387b&gt;] ? system_call_after_swapgs+0xc8/0x160 [16178.069707] [&lt;ffffffff8178386f&gt;] ? system_call_after_swapgs+0xbc/0x160 [16178.070768] [&lt;ffffffff8178387b&gt;] ? system_call_after_swapgs+0xc8/0x160 [16178.071896] [&lt;ffffffff8178386f&gt;] ? system_call_after_swapgs+0xbc/0x160 [16178.075009] [&lt;ffffffff81220eab&gt;] SyS_unlinkat+0x1b/0x40 [16178.076048] [&lt;ffffffff81783929&gt;] system_call_fastpath+0x16/0x1b [16178.077068] [&lt;ffffffff8178387b&gt;] ? system_call_after_swapgs+0xc8/0x160 </pre> </div></div> <p>It fails in multiple different tests too so it's not any particular testcase</p> <p>Andriy Skulysh (c17819@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/33718" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33718</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7558" title="niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)" class="issue-link" data-issue-key="LU-7558"><del>LU-7558</del></a> ptlrpc: connect vs import invalidate race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 994e02633ae373d454c20a9a6e5150d2d5312499</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33718/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33718/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7558" title="niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)" class="issue-link" data-issue-key="LU-7558"><del>LU-7558</del></a> ptlrpc: connect vs import invalidate race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b1827ff1da829ae5f320a417217757221eedda5f</p> <p>Landed for 2.13.0</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34290" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34290</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7558" title="niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)" class="issue-link" data-issue-key="LU-7558"><del>LU-7558</del></a> ptlrpc: connect vs import invalidate race<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 54f405709b1bef208d02468601a067a005dbb8e0</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34293" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34293</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7558" title="niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)" class="issue-link" data-issue-key="LU-7558"><del>LU-7558</del></a> ptlrpc: connect vs import invalidate race<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 9e4e28e244d0c2579ac5dc5f828be6dd77d708f6</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34290/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34290/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7558" title="niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)" class="issue-link" data-issue-key="LU-7558"><del>LU-7558</del></a> ptlrpc: connect vs import invalidate race<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: d5a51f0b718ecf6fca81e15c396e56141b62df6c</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34293/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34293/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7558" title="niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags &amp; MSGHDR_AT_SUPPORT ...)" class="issue-link" data-issue-key="LU-7558"><del>LU-7558</del></a> ptlrpc: connect vs import invalidate race<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 69d1e9805172c3e8da59ad99f470831951253695</p> Duplicate LU-9628 Related LU-5528 Development Rank 1|hzxvxb: Rank (Obsolete) 9223372036854775807 Severity