[LU-16876] lnet-selftest test_smoke: timeout ("2 batch in stopping")

• Status: Open
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: WC Triage
• Resolution: Unresolved
• Severity: 3

Description

This issue was created by maloo for eaujames <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/8852d1d0-2ea1-4ec4-adb9-a37e08fd7697

test_smoke failed with the following error:

Timeout occurred after 72 minutes, last suite running was lnet-selftest

Test session details:
clients: https://build.whamcloud.com/job/lustre-reviews/95331 - 4.18.0-425.3.1.el8.aarch64
servers: https://build.whamcloud.com/job/lustre-reviews/95331 - 4.18.0-425.3.1.el8_lustre.x86_64

Failed to stat on 2 nodes
[LNet Rates of s]
[R] Avg: 33       RPC/s Min: 33       RPC/s Max: 33       RPC/s
[W] Avg: 35       RPC/s Min: 35       RPC/s Max: 35       RPC/s
[LNet Bandwidth of s]
[R] Avg: 1.72     MiB/s Min: 1.72     MiB/s Max: 1.72     MiB/s 
[W] Avg: 2.22     MiB/s Min: 2.22     MiB/s Max: 2.22     MiB/s 
[LNet Rates of c]
[R] Avg: 34       RPC/s Min: 27       RPC/s Max: 41       RPC/s
[W] Avg: 34       RPC/s Min: 31       RPC/s Max: 37       RPC/s
[LNet Bandwidth of c]
[R] Avg: 3.53     MiB/s Min: 2.75     MiB/s Max: 4.32     MiB/s 
[W] Avg: 2.80     MiB/s Min: 2.75     MiB/s Max: 2.84     MiB/s 
Failed to stat on 3 nodes
[LNet Rates of c]
[R] Avg: 37       RPC/s Min: 35       RPC/s Max: 39       RPC/s
[W] Avg: 31       RPC/s Min: 29       RPC/s Max: 33       RPC/s
[LNet Bandwidth of c]
[R] Avg: 4.51     MiB/s Min: 4.41     MiB/s Max: 4.61     MiB/s 
[W] Avg: 1.97     MiB/s Min: 1.77     MiB/s Max: 2.16     MiB/s 
killing 19666 ...
5 batch in stopping
5 batch in stopping
2 batch in stopping
2 batch in stopping
....

Dmesg from all node:

[Mon Jun  5 17:26:41 2023] Lustre: DEBUG MARKER: trevis-108vm14.trevis.whamcloud.com: executing lst_setup
[Mon Jun  5 17:26:42 2023] LNet: 12305:0:(rpc.c:616:srpc_service_add_buffers()) waiting for adding buffer
[Mon Jun  5 17:26:42 2023] LNet: 12305:0:(rpc.c:616:srpc_service_add_buffers()) waiting for adding buffer
[Mon Jun  5 17:26:42 2023] LNet: 12305:0:(rpc.c:616:srpc_service_add_buffers()) waiting for adding buffer
...

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
lnet-selftest test_smoke - Timeout occurred after 72 minutes, last suite running was lnet-selftest