https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11931 Lustre <p>We have seen in a production system the following error which are causing clients to be evicted.</p> <p><span class="error">&#91;85895.120239&#93;</span> LNetError: 18866:0:(o2iblnd_cb.c:3271:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 8 seconds</p> <p><span class="error">&#91;85895.130310&#93;</span> LNetError: 18866:0:(o2iblnd_cb.c:3346:kiblnd_check_conns()) Timed out RDMA with 10.10.32.227@o2ib2 (51): c: 0, oc: 0, rc: 8</p> <p><span class="error">&#91;123887.254790&#93;</span> Lustre: MGS: haven't heard from client 51aa0ab0-3f34-cf7e-2fef-01e9ddcd4448 (at 732@gni4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff961d87b9a000, cur 1547261222 expire 1547261072 last 1547260995</p> <p>For our setup we have two back end file systems, F1 which is running 2.8.2 server back end and F2 which is running 2.11 server stack with ZFS (0.7.12). The clients are all running 2.11 cray clients. The LNet configuration is:</p> <p>F1 file system server backend with 2.8.2 stack, ldiskfs:</p> <p>    map_on_demand:0</p> <p>    concurrent_sends:0</p> <p>    peer_credits:8</p> <p>F2 file system server 2.11 (ZFS 0.7.12)</p> <p>    map_on_demand:1</p> <p>    concurrent_sends:63</p> <p>    peer_credits:8</p> <p>C3 (cray 2.11 router)</p> <p>   map_on_demand:0</p> <p>   concurrent_sends:16</p> <p>   peer_credits:8 (o2ib)</p> <p>   peer_credits:16 (gni). </p> <p>C4 (cray 2.11 router)</p> <p>   map_on_demand:0</p> <p>   concurrent_sends:63</p> <p>   peer_credits:8 (o2ib) </p> <p>   peer_credits:16 (gni)</p> <p>Currently the problems are only seen with 2.11 clients with the 2.11 file system. Since F1 is 2.8 and its peer credits are set to 8 this impacts the rest of the systems.</p> Cray CLE6 system running 2.11 clients with 2.11 servers. LU-11931

RDMA packets sent from client to MGS are timing out

Bug Critical Resolved Fixed Amir Shehata James A Simmons ORNL Tue, 5 Feb 2019 19:34:05 +0000 Wed, 22 May 2019 14:15:51 +0000 Sun, 21 Apr 2019 13:21:24 +0000 Lustre 2.11.0 Lustre 2.13.0 Lustre 2.12.1 0 13 <p>James</p> <p>So do I understand correctly that F2 is vanilla 2.11.0 code whereas C3 and C4 have patches applied by Cray (some of which may not have landed to master yet)?</p> <p>Amir</p> <p>What do you advise here?</p> <p>Peter</p> <p>James and I investigated this issue. We're currently suspecting it's due to peer_credits set to 8. There has been a change in 2.11:<br/> <a href="https://jira.whamcloud.com/browse/LU-10459" title="LBUG o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:" class="issue-link" data-issue-key="LU-10459"><del>LU-10459</del></a> lnd: throttle tx based on queue depth</p> <p>which throttles based on the queue depth as opposed to concurrent_sends which has been removed. In 2.8 concurrent_sends value would get set to 16. We're trying to do two things: 1) check to see if we can bump the queue depth to 16 (will require some interop testing with 2.8 on their test cluster) 2) see if we need to change the throttling code.</p> <p>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34200" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34200</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11931" title="RDMA packets sent from client to MGS are timing out " class="issue-link" data-issue-key="LU-11931"><del>LU-11931</del></a> lnd: relax throttling limit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 69f5e8e08a19f6fba884cf571de4b9a1307bed9a</p> <p>In our testing of the patch we did see this:</p> <p><span class="error">&#91;root@f2-util01 14:46:16&#93;</span><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=" class="user-hover" rel=""></a>#<br/> Message from syslogd@f2-util01.ncrc.gov at Mar 7 15:11:37 ...<br/> kernel:LNetError: 30311:0:(o2iblnd_cb.c:1019:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:<br/> Message from syslogd@f2-util01.ncrc.gov at Mar 7 15:11:37 ...<br/> kernel:LNetError: 30312:0:(o2iblnd_cb.c:1019:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:<br/> Message from syslogd@f2-util01.ncrc.gov at Mar 7 15:11:37 ...<br/> kernel:LNetError: 30312:0:(o2iblnd_cb.c:1019:kiblnd_check_sends_locked()) LBUG<br/> Message from syslogd@f2-util01.ncrc.gov at Mar 7 15:11:37 ...<br/> kernel:LNetError: 30305:0:(o2iblnd_cb.c:1019:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:<br/> Message from syslogd@f2-util01.ncrc.gov at Mar 7 15:11:37 ...<br/> kernel:LNetError: 30305:0:(o2iblnd_cb.c:1019:kiblnd_check_sends_locked()) LBUG<br/> Message from syslogd@f2-util01.ncrc.gov at Mar 7 15:11:37 ...<br/> kernel:LNetError: 30311:0:(o2iblnd_cb.c:1019:kiblnd_check_sends_locked()) LBUG<br/> Message from syslogd@f2-util01.ncrc.gov at Mar 7 15:11:37 ...<br/> kernel:Kernel panic - not syncing: LBUG</p> <p>Yes, this assert needs to change. Although, I'm considering now that it might be a good idea to bring back concurrent_sends. Initially, I was thinking that it's enough to limit the number of txs by queue depth, but it seems like in order to saturate the link you might want to increase the concurrent_sends to over the queue depth. This will lead to queued txs, but it might be necessary to make sure that we maximize the bandwidth</p> <p> </p> <p>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34396" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34396</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11931" title="RDMA packets sent from client to MGS are timing out " class="issue-link" data-issue-key="LU-11931"><del>LU-11931</del></a> lnd: bring back concurrent_sends<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2dead91d20b77e6c279aa1ca048b53d6f5617b10</p> <p>We have released this patch into our production server system and it has resolved the peer credit starvation issues on MGS that was causing client evictions. The work around before the patch was to remove a batch of client nodes until the evictions stopped. Now with the patch in production we have all the clients back in use. Please consider landing this for 2.12 LTS.</p> <p>We'll land it to b2_12 as soon as it's landed to master. ATM it's the -1 review from you gating that. Are you willing to reconsider that -1 in light of the success of the patch or did you actually revise the patch as you have suggested before applying it in production?</p> <p>Actually their are two patches. I nicked the one patch but I like the other patch that restored the concurrent_send functionality. Also the other patch is what we ended up running in production</p> <p>Ah I see. So then we'll press to get  34396 landed and 34200 should probably move to being tracked under a new Jira reference. That way it'll be simplest to ensure that we get the desired patch into 2.12.1</p> <p>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34646" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34646</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11931" title="RDMA packets sent from client to MGS are timing out " class="issue-link" data-issue-key="LU-11931"><del>LU-11931</del></a> lnd: bring back concurrent_sends<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: f08724454d919717e8a70555cf1194acddc731ad</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34396/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34396/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11931" title="RDMA packets sent from client to MGS are timing out " class="issue-link" data-issue-key="LU-11931"><del>LU-11931</del></a> lnd: bring back concurrent_sends<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 83e45ead69babfb2909a3157f054fcd8fdf33360</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34646/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34646/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11931" title="RDMA packets sent from client to MGS are timing out " class="issue-link" data-issue-key="LU-11931"><del>LU-11931</del></a> lnd: bring back concurrent_sends<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 056fe83188f0a24de9e27248a7574c8fae867163</p> <p>ok. Main patch landed for 2.13. 34200 should be tracked under a new Jira ticket</p> Related LU-10291 LU-12279 Development Rank 1|i00b1r: Rank (Obsolete) 9223372036854775807 Severity