[LU-8485] workqueue overflows with mlx5 on power8 platforms.

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Environment: Power8 client nodes running RHEL7.2 with Mellanox OFED 3.2-1.04
• Severity: 3
• Assignee: Doug Oucharek (Inactive)
• Reporter: James A Simmons

Description

Currently in my testing on the Power8 platform I from time to time see the following errors on the clients and the lustre becomes unusable.

[ 3499.198051] mlx5_warn:mlx5_0:begin_wqe:4013:(pid 7712): work queue overflow
[ 3499.198176] mlx5_warn:mlx5_0:mlx5_ib_post_send:4112:(pid 7712): Failed to prepare WQE
[ 3499.198209] mlx5_warn:mlx5_0:begin_wqe:4013:(pid 7715): work queue overflow
[ 3499.198240] LustreError: 7712:0:(events.c:203:client_bulk_callback()) event type 1, status -12, desc c000001772778c00
[ 3499.198428] mlx5_warn:mlx5_0:mlx5_ib_post_send:4112:(pid 7715): Failed to prepare WQE
[ 3499.198527] LustreError: 7715:0:(events.c:203:client_bulk_callback()) event type 1, status -12, desc c000000788600c00
[ 3499.199804] LustreError: 7713:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000e27e06800
[ 3499.199928] LustreError: 7714:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000788602200
[ 3499.200740] LustreError: 7712:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c00000077cec7400
[ 3499.201667] LustreError: 7715:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c00000039da2f400
[ 3499.202216] LustreError: 7715:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000780129c00
[ 3499.202422] LustreError: 7713:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000e270c3000
[ 3499.202642] LustreError: 7715:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000001b98441800
[ 3499.202864] LustreError: 7712:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000c6d9fd600
[ 3499.203091] LustreError: 7714:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000dd0309200
[ 3499.203942] LustreError: 7713:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc c000000e27e06200
[ 3499.558222] LNet: 7659:0:(o2iblnd_cb.c:1360:kiblnd_reconnect_peer()) Abort reconnection of 10.37.248.77@o2ib1: connected
[ 3499.558317] LNet: 7659:0:(o2iblnd_cb.c:1360:kiblnd_reconnect_peer()) Skipped 4 previous similar messages

Issue Links

• is related to: LU-6387 Add Power8 support to Lustre

Comments

[James A Simmons]

Worked with Doug to track down the issue reported in this ticket. The main problem was due to the IBLND_SEND_WRS macro in o2iblnd not creating a deep enough queue. It was using the local frag size (16) but it needs to assume the worst case of working with a external node (256) so the queue ended up too small and it would be easily overrun. The latest patch http://review.whamcloud.com/21304 should address these problems.

Besides the fixes in the 21304 patch the queue problems existed in that it they were to small. The solution to that was to reduce the concurrent_sends from 63 down to 31 and Lustre started to function in all my test cases.

[Doug Oucharek (Inactive)]

After a debugging session, we seem to have tracked down a few problems with the 21304 patch and are close to have a version which works without problems (i.e. overflows). As such, I'm going to mark this ticket resolved and leave the final changes to 21304 in its ticket LU-7650.

[James A Simmons]

I updated the patch and the problem still exist. I will push a new version of the 21304 patch.

[Doug Oucharek (Inactive)]

Please see my comments on http://review.whamcloud.com/21304/. A solution to this ticket may come about when addressing my comments.

[James A Simmons]

Here is a lctl dump from my power8 client nodes. For the server side we are using standard x86_64 platforms which is why we are having issues.

[James A Simmons]

Yes they are on. It will take me some time to get any lctl debug logs since this problem happens randomly.

[Doug Oucharek (Inactive)]

Are NETERRORS turned on? I'm curious to see if o2iblnd has any messages for us to help.

[James A Simmons]

With since without patch 21304 ko2iblnd doesn't work on Power8 platforms. A small bug exist in the patch that I submitted but I have a version locally that appears to work.

[Doug Oucharek (Inactive)]

James, is this failure with or without your patch: http://review.whamcloud.com/21304/?

[Peter Jones]

Doug is looking into this