https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8485 Lustre <p>Currently in my testing on the Power8 platform I from time to time see the following errors on the clients and the lustre becomes unusable.</p> <p>[ 3499.198051] mlx5_warn:mlx5_0:begin_wqe:4013:(pid 7712): work queue overflow<br/> [ 3499.198176] mlx5_warn:mlx5_0:mlx5_ib_post_send:4112:(pid 7712): Failed to prepare WQE<br/> [ 3499.198209] mlx5_warn:mlx5_0:begin_wqe:4013:(pid 7715): work queue overflow<br/> [ 3499.198240] LustreError: 7712:0:(events.c:203:client_bulk_callback()) event type 1, status -12, desc c000001772778c00<br/> [ 3499.198428] mlx5_warn:mlx5_0:mlx5_ib_post_send:4112:(pid 7715): Failed to prepare WQE<br/> [ 3499.198527] LustreError: 7715:0:(events.c:203:client_bulk_callback()) event type 1, status -12, desc c000000788600c00<br/> [ 3499.199804] LustreError: 7713:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000e27e06800<br/> [ 3499.199928] LustreError: 7714:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000788602200<br/> [ 3499.200740] LustreError: 7712:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c00000077cec7400<br/> [ 3499.201667] LustreError: 7715:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c00000039da2f400<br/> [ 3499.202216] LustreError: 7715:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000780129c00<br/> [ 3499.202422] LustreError: 7713:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000e270c3000<br/> [ 3499.202642] LustreError: 7715:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000001b98441800<br/> [ 3499.202864] LustreError: 7712:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000c6d9fd600<br/> [ 3499.203091] LustreError: 7714:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000dd0309200<br/> [ 3499.203942] LustreError: 7713:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000e27e06200<br/> [ 3499.558222] LNet: 7659:0:(o2iblnd_cb.c:1360:kiblnd_reconnect_peer()) Abort reconnection of 10.37.248.77@o2ib1: connected<br/> [ 3499.558317] LNet: 7659:0:(o2iblnd_cb.c:1360:kiblnd_reconnect_peer()) Skipped 4 previous similar messages</p> Power8 client nodes running RHEL7.2 with Mellanox OFED 3.2-1.04 LU-8485

workqueue overflows with mlx5 on power8 platforms.

Bug Critical Resolved Fixed Doug Oucharek James A Simmons Mon, 8 Aug 2016 21:50:24 +0000 Fri, 12 Aug 2016 21:32:45 +0000 Fri, 12 Aug 2016 21:13:59 +0000 Lustre 2.8.0 Lustre 2.9.0 0 4 <p>Doug is looking into this</p> <p>James, is this failure with or without your patch: <a href="http://review.whamcloud.com/21304/?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21304/?</a></p> <p>With since without patch 21304 ko2iblnd doesn't work on Power8 platforms. A small bug exist in the patch that I submitted but I have a version locally that appears to work.</p> <p>Are NETERRORS turned on? I'm curious to see if o2iblnd has any messages for us to help.</p> <p>Yes they are on. It will take me some time to get any lctl debug logs since this problem happens randomly.</p> <p>Here is a lctl dump from my power8 client nodes. For the server side we are using standard x86_64 platforms which is why we are having issues.</p> <p>Please see my comments on <a href="http://review.whamcloud.com/21304/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21304/</a>. A solution to this ticket may come about when addressing my comments.</p> <p>I updated the patch and the problem still exist. I will push a new version of the 21304 patch.</p> <p>After a debugging session, we seem to have tracked down a few problems with the 21304 patch and are close to have a version which works without problems (i.e. overflows). As such, I'm going to mark this ticket resolved and leave the final changes to 21304 in its ticket <a href="https://jira.whamcloud.com/browse/LU-7650" title="ko2iblnd map_on_demand can&#39;t negotitate when page sizes are different between nodes." class="issue-link" data-issue-key="LU-7650"><del>LU-7650</del></a>.</p> <p>Worked with Doug to track down the issue reported in this ticket. The main problem was due to the IBLND_SEND_WRS macro in o2iblnd not creating a deep enough queue. It was using the local frag size (16) but it needs to assume the worst case of working with a external node (256) so the queue ended up too small and it would be easily overrun. The latest patch <a href="http://review.whamcloud.com/21304" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21304</a> should address these problems.</p> <p>Besides the fixes in the 21304 patch the queue problems existed in that it they were to small. The solution to that was to reduce the concurrent_sends from 63 down to 31 and Lustre started to function in all my test cases.</p> Related LU-6387 Development Rank 1|hzyjxb: Rank (Obsolete) 9223372036854775807 Severity