-
Task
-
Resolution: Fixed
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Since September, most lnet-selftest test_smoke failures on review patches have had a VM from onyx-75 in the session (onyx-75vm1..13, 10.240.26.76-88). The VMs on that host answer LNet RPCs 10-40 seconds late, which is long enough to fail the test either way it can fail ("lst Error found", or the "N batch in stopping" suite timeout).
What the data shows (Maloo, via Rainshadow; lustre-reviews lnet-selftest failures 2026-08-24 to 2026-10-09):
- 28 of 29 failed sessions include an onyx-75 VM. onyx-75 VMs are about 733 of 236,900 node-sessions in that window (0.3%).
- In those failures, lst stat (10 s RPC timeout) reports "Failed to stat" on the onyx-75 nodes in 27-28 of its 29 sample intervals. The console debug logs show their replies arriving 10-37 s late (median about 14 s). In 79 of 80 sampled passing runs on other hosts that count is 0.
- BRW RPCs to the same nodes pass the 64 s selftest RPC timeout ("Client RPC expired: service 11 ... timeout 64").
- The VMs' own dmesg shows nothing: no OOM, soft lockup, hung task, NIC or time-sync messages. That points at the host (load/oversubscription, network, NIC or bridge) rather than the guests.
It is the host, not the distro or ARM:
- 2026-10-07 to 10-10: 11 of 13 lnet-selftest runs with an onyx-75 VM show the slow-reply pattern (only 2 of those 13 failed; the rest passed slowly). They span el8.10, el9.7 and el9.8 servers and five client distros, including aarch64.
- Same days, 51 runs with RHEL 8.10 servers on other hosts show none of it.
- The ARM review groups look much worse than x86 (37/1607 failed suite runs vs 12/9889), but in every ARM failure the x86 servers were onyx-75 VMs; the aarch64 clients were never the slow side.
It started recently:
- June 2026: 13 of 14 lnet-selftest runs with an onyx-75 VM were clean, at 300-1300 MiB/s. Now the same runs often move 3-100 MiB/s.
- lnet-selftest failures on master/master-next/reviews per month: July 13, August 11, September 42 (28 with onyx-75), October 18 in the first ten days (17 with onyx-75).
- Smaller bursts with the same pattern on other hosts: onyx-154/159 on 2026-09-17, onyx-156..159 in late August.
Example sessions (onyx-75 role in brackets):
- https://testing.whamcloud.com/test_sessions/59f13453-72fc-4693-8db1-cade120f0db8 - 2026-10-08, review-ldiskfs-arm, suite timeout (servers onyx-75vm10, vm11)
- https://testing.whamcloud.com/test_sessions/26cd30dd-ed18-4086-b084-960b185e92d5 - 2026-10-09, review-dne-part-2, "lst Error found" (servers onyx-75vm9, vm12)
- https://testing.whamcloud.com/test_sessions/e9e2e75a-1d1b-464f-a86c-f4de4f5f20f2 - 2026-10-08, review-ldiskfs, passed but at 5 MiB/s (clients onyx-75vm10, vm11)
- https://testing.whamcloud.com/test_sessions/818a6ef1-c313-47fb-acd2-b48d33df7958 - 2026-06-22, review-dne-zfs-part-2, healthy for comparison: 1290 MiB/s, all five VMs on onyx-75
Request: please check onyx-75 (host load and VM packing, NIC/bridge errors and drops, the network path), and take it out of rotation until it is fixed if that is easier. lnet-selftest is where this shows most clearly, but any suite with a VM on that host is presumably running slowly too.