https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6748 Lustre <p>While testing the last pre-2.8 code I noticed heavy client reconnects to OSS servers. The error on the client side was:</p> <p>Lustre: sultan-OST0008-osc-ffff8803ea302800: Connection to sultan-OST0008 (at 10.37.248.69@o2ib1) was lost; in progress operations using this service will wait for recovery to complete<br/> Lustre: Skipped 55 previous similar messages<br/> Lustre: 5355:0:(client.c:2009:ptlrpc_expire_one_request()) Skipped 61 previous similar messages<br/> Lustre: 5350:0:(client.c:2009:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1434742560/real 1434742560&#93;</span> req@ffff8803c23fb6c0 x1504421695570504/t0(0) o8-&gt;sultan-OST0023-osc-ffff8803ea302800@10.37.248.72@o2ib1:28/4 lens 520/544 e 0 to 1 dl 1434742568 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Lustre: 5350:0:(client.c:2009:ptlrpc_expire_one_request()) Skipped 7 previous similar messages<br/> Lustre: sultan-OST0000-osc-ffff8803ea302800: Connection restored to sultan-OST0000 (at 10.37.248.69@o2ib1)<br/> Lustre: Skipped 27 previous similar messages<br/> Lustre: 5356:0:(client.c:2009:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1434742782/real 1434742782&#93;</span> req@ffff8803c1b639c0 x1504421695572244/t0(0) o400-&gt;sultan-OST0034-osc-ffff8803ea302800@10.37.248.69@o2ib1:28/4 lens 224/224 e 0 to 1 dl 1434742789 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Lustre: sultan-OST0000-osc-ffff8803ea302800: Connection to sultan-OST0000 (at 10.37.248.69@o2ib1) was lost; in progress operations using this service will wait for recovery to complete<br/> Lustre: Skipped 41 previous similar messages<br/> Lustre: 5356:0:(client.c:2009:ptlrpc_expire_one_request()) Skipped 73 previous similar messages<br/> Lustre: sultan-OST0000-osc-ffff8803ea302800: Connection restored to sultan-OST0000 (at 10.37.248.69@o2ib1)<br/> Lustre: Skipped 41 previous similar messages<br/> Lustre: sultan-OST0003-osc-ffff8803ea302800: Connection restored to sultan-OST0003 (at 10.37.248.72@o2ib1)<br/> Lustre: Skipped 27 previous similar messages</p> <p>and the messages seen on the OSS side are:</p> <p>20639.820176] Lustre: sultan-OST0008: Client 57c62113-31f1-f463-ffeb-9d0c7541279d (at 26@gni1) reconnecting<br/> <span class="error">&#91;20639.829910&#93;</span> Lustre: Skipped 20 previous similar messages<br/> <span class="error">&#91;20676.881745&#93;</span> Lustre: sultan-OST000c: Client 57c62113-31f1-f463-ffeb-9d0c7541279d (at 26@gni1) reconnecting<br/> <span class="error">&#91;20676.891462&#93;</span> Lustre: Skipped 29 previous similar messages<br/> <span class="error">&#91;20868.910972&#93;</span> Lustre: sultan-OST0004: Client 57c62113-31f1-f463-ffeb-9d0c7541279d (at 26@gni1) reconnecting<br/> <span class="error">&#91;20868.920682&#93;</span> Lustre: Skipped 23 previous similar messages<br/> <span class="error">&#91;20906.993360&#93;</span> Lustre: sultan-OST0000: Client 57c62113-31f1-f463-ffeb-9d0c7541279d (at 26@gni1) reconnecting<br/> <span class="error">&#91;20906.993364&#93;</span> Lustre: sultan-OST0004: Client 57c62113-31f1-f463-ffeb-9d0c7541279d (at 26@gni1) reconnecting<br/> <span class="error">&#91;20906.993368&#93;</span> Lustre: Skipped 17 previous similar messages<br/> <span class="error">&#91;20907.018191&#93;</span> Lustre: Skipped 11 previous similar messages</p> <p>This occured when I ran a file per process IOR job across 20 nodes with 32 threads per client. </p> LU-6748

excessive client reconnect to OSS servers under heavy IO work load.

Bug Critical Resolved Duplicate Jinshan Xiong James A Simmons Fri, 19 Jun 2015 20:55:20 +0000 Mon, 31 Aug 2015 16:46:42 +0000 Mon, 31 Aug 2015 16:46:26 +0000 Lustre 2.8.0 0 3 <p>Uploaded logs to ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-6748" title="excessive client reconnect to OSS servers under heavy IO work load." class="issue-link" data-issue-key="LU-6748"><del>LU-6748</del></a>/*</p> <p>the client was experiencing slow reply, but OSS even has ever sent early replies.</p> <p>Here are some suspicious messages:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;node_health:5.1&gt; APID:831 (xtcheckhealth) WARNING: Advanced_features and anyapid check are both configured on. Application test could falsely mark nodes unhealthy. &lt;node_health:5.1&gt; RESID:3043 (xtcheckhealth) WARNING: Advanced_features and anyapid check are both configured on. Application test could falsely mark nodes unhealthy. </pre> </div></div> <p>I believe they are from GNI lnd, did this message imply anything?</p> <p>That is the Cray health check warning the file system is sick. The message comes from a user land utility.</p> <p>When did you notice the regression? Can you identify which patches, or a range of date when the patches were landed?</p> <p>From your description, it looks like this is related to ldlm lock handling. Please check the LRU size on the client side and how many locks are cached in the LRU when the problem is reproduced. If possible, please dump the locks so that we can make further investigation.</p> <p>Just as I thought. I found the source of the regression. I had map_on_demand set on the OSS servers. I unmounted the file system and restarted the o2iblnd layer without map_on_demand set and I stopped seeing reconnecting issues. So the map_on_demand is a problem on normal systems as well.</p> <p>We can close this as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-6723" title="Setting map_on_demand for o2iblnd driver prevents lustre bring up." class="issue-link" data-issue-key="LU-6723"><del>LU-6723</del></a></p> Duplicate LU-6723 Related LU-6723 Development Rank 1|hzxg87: Rank (Obsolete) 9223372036854775807 Severity