https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13763 Lustre <p>Console log messages following this pattern, repeatedly, for several days:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 67801:0:(import.c:361:ptlrpc_invalidate_import()) lsrza-OST0000_UUID: rc = -110 waiting for callback (1 != 0) LustreError: 67801:0:(import.c:387:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff8c3eb65f7500 x1669124751850560/t0(0) o4-&gt;lsrza-OST0000-osc-ffff8c44c608a000@172.21.3.5@o2ib700:6/4 lens 488/448 e 2 to 0 dl 1592847228 ref 1 fl Interpret:E/0/ffffffff rc -5/-1 LustreError: 67801:0:(import.c:401:ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out. </pre> </div></div> <p>Note that the number in the parentheses is 0. This refers to imp-&gt;imp_unregistering, an atomic variable that looks like it is intended to track the number of RPC buffers we ware waiting for the underlying network to unregister so know that no data will be lost. But there is still one RPC on the sending list, so why is imp_unregistering 0?</p> TOSS 3.6-3 / RH78 <br/> in-kernel OFED <br/> 3.10.0-1127.8.2.1chaos.ch6.x86_64 <br/> lustre-2.12.4_6.chaos-1.ch6.x86_64 <br/> LU-13763

ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out.

Bug Minor Resolved Fixed Mikhail Pershin Olaf Faaland llnl Wed, 8 Jul 2020 19:27:24 +0000 Thu, 26 Nov 2020 18:58:25 +0000 Sat, 12 Sep 2020 15:56:40 +0000 Lustre 2.12.4 Lustre 2.14.0 Lustre 2.12.6 0 5 <p>Similar to <a href="https://jira.whamcloud.com/browse/LU-13303" title="(import.c:361:ptlrpc_invalidate_import()) nbp1-OST0016_UUID: rc = -110 waiting for callback (1 != 0)" class="issue-link" data-issue-key="LU-13303"><del>LU-13303</del></a> but in this case imp_unregistering is 0.</p> <p>For my tracking, my internal issue is TOSS4833</p> <p>Mike</p> <p>Could you please advise</p> <p>Thanks</p> <p>Peter</p> <p>The node where I saw this is still in this state.</p> <p>While looking into <a href="https://jira.whamcloud.com/browse/LU-13766" title="tgt_grant_check() lsrza-OST000a: cli dfdf1aff-07d9-53b3-5632-c18a78027eb2 claims 1703936 GRANT, real grant 0" class="issue-link" data-issue-key="LU-13766"><del>LU-13766</del></a>, I found that the node with this symptom also has a cur_grant_bytes that is weirdly large; I wonder if it underflowed. I don't know if this is related or not, but it seems an unlikely coincidence, so here it is.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@rzgenie28:~]# lctl get_param -n osc.*OST0000*.cur_grant_bytes 18446744073707847680 </pre> </div></div> <p>I've attached console and debug logs.</p> <p>Thanks, Olaf. I am checking logs right now.</p> <p>In osc_init_grant(), cl_avail_grant() will underflow if <br/> cl_reserved_grant + (cl_dirty_grant OR cl_dirty_pages&lt;&lt;PAGE_SHIFT) &gt; ocd_grant</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> cli-&gt;cl_avail_grant = ocd-&gt;ocd_grant; if (cli-&gt;cl_import-&gt;imp_state != LUSTRE_IMP_EVICTED) { cli-&gt;cl_avail_grant -= cli-&gt;cl_reserved_grant; if (OCD_HAS_FLAG(ocd, GRANT_PARAM)) cli-&gt;cl_avail_grant -= cli-&gt;cl_dirty_grant; else cli-&gt;cl_avail_grant -= cli-&gt;cl_dirty_pages &lt;&lt; PAGE_SHIFT; } </pre> </div></div> <p>I don't know if there's something to prevent that underflow elsewhere.</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39380" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39380</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13763" title="ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out." class="issue-link" data-issue-key="LU-13763"><del>LU-13763</del></a> osc: don't allow negative grants<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: d707c6ee3a926b06ebb0c2648cb5f1e5a1aaf2d7</p> <p>Olaf, I've made patch to prevent that underflow. It is worth to do in any case because <tt>ocd_grant</tt> is received from server so it shouldn't be trusted blindly to be always greater than local consumed grants.</p> <p>E.g. in conjunction with <a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> that looks as real case.</p> <p>Mikhail,</p> <p>I have the node in this state drained. Should I keep it that way in case you want me to gather information from it, or should I go ahead and crash it and put it back into service? I won't be able to send you the crash dump, but I could extract information from the dump for you, potentially (although my crash skills are not great).</p> <p>Thanks</p> <p>Hi Mikhail,<br/> Can I bounce that node?<br/> Do you have any thoughts on whether the stuck import was really related to the grant underflow?<br/> Do you have any idea why imp_unregistering was 0 when there was still one RPC on the sending list?<br/> Thanks!</p> <p>Olaf, I have no good idea about what to get from that node right now, so you can bounce it. I don't see how grant underflow could occur due to this particular situation - for that server should consider that client has less grants than it really has already. I'd say that such situation could occur due to DIO grants problem and then cause grants underflow later. Maybe there are other scenarios exist. As for imp_unregistering 0 I think it is OK if request is in sending list - it is waiting for reply still, so is not unregistered yet and imp counter is still 0.</p> <p>I rebased the patch as it was too old to be retested.</p> <p>The patch passes testing now. I posted a comment listing all the failures. It never failed a test twice, every test that failed passed when that test group was re-tested, and the failures didn't look to me like grant accounting problems.</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39827" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39827</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13763" title="ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out." class="issue-link" data-issue-key="LU-13763"><del>LU-13763</del></a> osc: don't allow negative grants<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bd9497c1690b9c4d2358a287277d1d67864d6735</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39827/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39827/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13763" title="ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out." class="issue-link" data-issue-key="LU-13763"><del>LU-13763</del></a> osc: don't allow negative grants<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e05ccafd6ee214895d01efbb13a3757e3625a859</p> <p>Landed for 2.14</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39380/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39380/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13763" title="ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out." class="issue-link" data-issue-key="LU-13763"><del>LU-13763</del></a> osc: don't allow negative grants<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: f96aa90548f062e95d2ef4c9ea978ba0e08aae19</p> Duplicate Related Development Rank 1|i014on: Rank (Obsolete) 9223372036854775807 Severity