https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6441 Lustre <p>As part of acceptance testing, an lnet router node was deliberately crashed (via sysrq-trigger). Following the crash, a set of OSTS nodes started reporting problems, hung threads, timeouts, clients continually losing connection, reconnecting, etc. Nodes are held on ptlrpc_abort_bulk() function.</p> <p>Logs shows messages like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Feb 25 21:17:15 somenode kernel: LustreError: 32278:0:(service.c:3214:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 16940s Feb 25 21:17:15 somenode kernel: LustreError: 32278:0:(service.c:3214:ptlrpc_svcpt_health_check()) Skipped 5 previous similar messages Feb 25 21:20:31 somenode kernel: Lustre: 106418:0:(niobuf.c:282:ptlrpc_abort_bulk()) Unexpectedly <span class="code-object">long</span> timeout: desc ffff8802e3970000 </pre> </div></div> <p>The system uses a fine-grained routing configuration. It was one of the 5 routers in this set that was killed. The expectation is minimal disruption for clients, the other 4 routers are still functioning, clients should detect that one is down and send traffic to the other 4 nodes. </p> Servers runs 2.5.1, clients run Lustre 2.5.1 LU-6441

OST problems following router node crash, inactive threads, clients continuously reconnecting

Bug Major Resolved Fixed Emoly Liu Artem Blagodarenko patch Wed, 8 Apr 2015 09:03:47 +0000 Wed, 22 Jun 2022 20:40:19 +0000 Fri, 1 May 2015 11:32:18 +0000 Lustre 2.5.1 Lustre 2.8.0 Lustre 2.9.0 1 9 <p>Artem Blagodarenko (artem_blagodarenko@xyratex.com) uploaded a new patch: <a href="http://review.whamcloud.com/14399" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14399</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6441" title="OST problems following router node crash, inactive threads, clients continuously reconnecting" class="issue-link" data-issue-key="LU-6441"><del>LU-6441</del></a> ptlrpc: ptlrpc_bulk_abort unlink all entries in bd_mds<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0feb07fa9852dfeb8fa68ea80587cb7f11a7ab04</p> <p>This bug happened when 4MB io is enabled. We noticed it already on two clusters. I believe this patch is important for somebody who going to use 4MB io.</p> <p>Agreed re 4MB rpc, also doesn't require an LNET router crash/panic, can easily reproduce it with a similar trigger such as pulling IB cables (or whatever network you are using) on the clients while the clients are doing I/O to the filesystem.</p> <p>Patching the server with the above patch I can confirm resolves the problem. Have done this on site at ANU/NCI on the available test kit, i.e. was able to reproduce, patch the server and install then spent a day and a half trying to reproduce (when I could hit this 2/3 attempts on a server without the patch). </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14399/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14399/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6441" title="OST problems following router node crash, inactive threads, clients continuously reconnecting" class="issue-link" data-issue-key="LU-6441"><del>LU-6441</del></a> ptlrpc: ptlrpc_bulk_abort unlink all entries in bd_mds<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0a6470219a8602d7a56fe1c5171dba4a42244738</p> <p>Landed for 2.8</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22403" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22403</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6441" title="OST problems following router node crash, inactive threads, clients continuously reconnecting" class="issue-link" data-issue-key="LU-6441"><del>LU-6441</del></a> ptlrpc: fix the problem of the patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6bfc8faac211e5c2dd5a369ac23438565d3d16c0</p> <p>Jinshan</p> <p>It would be better to open a new ticket and link to this one with any changes needed to the original patch</p> <p>Peter</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22403/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22403/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6441" title="OST problems following router node crash, inactive threads, clients continuously reconnecting" class="issue-link" data-issue-key="LU-6441"><del>LU-6441</del></a> ptlrpc: fix sanity 224c for different RPC sizes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6cde14a5df781ae29da88f98a2559eb4342fe1f3</p> Blocker Duplicate Related LU-6573 LU-6808 Development Rank 1|hzxac7: Rank (Obsolete) 9223372036854775807 Severity