MPI jobs in Autotest sessions fail to start when the two client VMs are on different physical hosts. OpenMPI uses the UCX PML, and the UCX TCP connections between the MPI processes on the two clients time out:

[onyx-140vm20:2804353:0] sock.c:334 UCX ERROR connect(fd=37, dest_addr=10.240.26.129:51651) failed: Connection timed out
[onyx-140vm20.onyx.whamcloud.com:2804352] pml_ucx.c:419 Error: ucp_ep_create(proc=2) failed: Destination is unreachable
[onyx-140vm20.onyx.whamcloud.com:2804352] pml_ucx.c:472 Error: Failed to resolve UCX endpoint for rank 2
[onyx-140vm20:2804353] *** An error occurred in MPI_Comm_split_type
[onyx-140vm20:2804353] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,

The test program never runs, so the test fails as "ior failed! 1", "metabench failed! 1" and so on, and every MPI test in the session fails the same way. Most of these are currently linked to LU-19659, whose own symptom (wrong data from ior over NFS) is a different failure.

The mpirun options the tests pass, "--mca btl tcp,self --mca btl_tcp_if_include eth0", do not constrain this: they configure the TCP BTL, and UCX is not the BTL. The connects go to both the 10.240.x (eth0) and the 172.16.x addresses of the other client.

Placement is the cause. Every one of the 9 UCX failures of parallel-scale-nfsv3 test_iorssf between 2026-08-15 and 2026-09-19 had its two clients on different physical hosts:

All 34 passing runs of the same test on lustre-master over the last 60 days had both clients on the same host (onyx-145vm1 and onyx-145vm2, trevis-152vm24 and trevis-152vm9, and so on). So MPI across hosts appears to fail every time; it shows up as a rare failure only because the two clients are usually placed together. LNet between nodes on different hosts works in the same sessions (the servers are on other hosts), so it is the MPI processes' connections to ephemeral ports that do not get through.

Possible fixes, in Autotest rather than Lustre:

• keep the client VMs of a session on one physical host, or
• allow the inter-host traffic the client VMs need (ephemeral TCP ports between clients), or
• restrict UCX to a working interface (UCX_NET_DEVICES=eth0) or select the TCP BTL (--mca pml ob1) in MPIRUN_OPTIONS. Not tested; if ephemeral ports between hosts are blocked, the TCP BTL may hit the same wall.

Related: LU-16658 (2023, performance-sanity, UCX "Destination is unreachable"; resolved by disabling the test).