[LU-14733] brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.12.8, Lustre 2.15.0
• Labels: LTS12 llnl
• Environment: lustre-2.12.6_9.llnl client ; kernel-4.18.0-305.0.0.1toss.t4.x86_64 ; RHEL84
• Severity: 3
• Assignee: Serguei Smirnov
• Reporter: Olaf Faaland

Description

lnet_selftest fails between two nodes over Omnipath

dk.opal63.llnl.gov.7:00000001:00020000:43.0:1622598261.714620:0:129525:0:(brw_test.c:415:brw_bulk_ready()) BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103 

Bulk transfers work over Infiniband (although in that test 1 of the nodes was RHEL 7.9 and an earlier Lustre patch stack).  Bulk transfers also work over tcp using ksocklnd.

lctl pings work fine between the same two nodes.

mpibench and other MPI applications also work fine over Omnipath between two nodes.

See https://github.com/LLNL/lustre/releases/tag/2.12.6_9.llnl for the patch stack

Issue Links

• is duplicated by: LU-13976 duplicate IB_WR_LOCAL_INV causing ice driver failure (RoCE/iWarp)
• is related to: LU-15116 crash when writing files in parallel on LTS lustre version

Comments

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" merged in patch https://review.whamcloud.com/44296/
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: b2_14
Current Patch Set:
Commit: 29da7cba3e7b3461d895010c7f7284b9649aba52

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" merged in patch https://review.whamcloud.com/44295/
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: b2_14
Current Patch Set:
Commit: 380be07fcca1f76564d1f29e58f2d8d5f8f530c8

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44217/
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 96d7dcf4e773e6026a590e4596ef30ac8a4a5061

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44216/
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 173d60a3274d19bc1d9811b6e1b09aac2b25f221

[Peter Jones]

Landed for 2.15

[Olaf Faaland]

Mike, Serguei, this works on our test system

[Gerrit Updater]

Gian-Carlo DeFazio ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44296
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: b2_14
Current Patch Set: 1
Commit: 1105bfbfad5002bc673ff568c0720bcebc5d095a

[Gerrit Updater]

Gian-Carlo DeFazio ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44295
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: b2_14
Current Patch Set: 1
Commit: 8db93a26f7e99f140ccd4a0fd3e35f4e9f71b8ec

[Mike Marciniszyn]

05:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]

I was able to run lnet_selftest on a vanilla RH 8.4 install:

[root@ioperf-05 ~]# ./lnet_wrapper_read
LST_SESSION = 6780
SESSION: lstread FEATURES: 1 TIMEOUT: 300 FORCE: No
192.168.25.9@o2ib1 are added to session
192.168.25.10@o2ib1 are added to session
Test was added successfully
bulk_read is running now
Capturing statistics for 30 secs [LNet Rates of lfrom]
[R] Avg: 21275    RPC/s Min: 21275    RPC/s Max: 21275    RPC/s
[W] Avg: 10638    RPC/s Min: 10638    RPC/s Max: 10638    RPC/s
[LNet Bandwidth of lfrom]
[R] Avg: 10639.30 MiB/s Min: 10639.30 MiB/s Max: 10639.30 MiB/s
[W] Avg: 1.62     MiB/s Min: 1.62     MiB/s Max: 1.62     MiB/s
[LNet Rates of lto]
[R] Avg: 10637    RPC/s Min: 10637    RPC/s Max: 10637    RPC/s
[W] Avg: 21275    RPC/s Min: 21275    RPC/s Max: 21275    RPC/s
[LNet Bandwidth of lto]
[R] Avg: 1.62     MiB/s Min: 1.62     MiB/s Max: 1.62     MiB/s
[W] Avg: 10639.10 MiB/s Min: 10639.10 MiB/s Max: 10639.10 MiB/s
[LNet Rates of lfrom]
[R] Avg: 21294    RPC/s Min: 21294    RPC/s Max: 21294    RPC/s
[W] Avg: 10647    RPC/s Min: 10647    RPC/s Max: 10647    RPC/s
[LNet Bandwidth of lfrom]
[R] Avg: 10647.90 MiB/s Min: 10647.90 MiB/s Max: 10647.90 MiB/s
[W] Avg: 1.62     MiB/s Min: 1.62     MiB/s Max: 1.62     MiB/s
[LNet Rates of lto]
[R] Avg: 10647    RPC/s Min: 10647    RPC/s Max: 10647    RPC/s
[W] Avg: 21294    RPC/s Min: 21294    RPC/s Max: 21294    RPC/s
[LNet Bandwidth of lto]
[R] Avg: 1.62     MiB/s Min: 1.62     MiB/s Max: 1.62     MiB/s
[W] Avg: 10647.90 MiB/s Min: 10647.90 MiB/s Max: 10647.90 MiB/s
[LNet Rates of lfrom]
[R] Avg: 21275    RPC/s Min: 21275    RPC/s Max: 21275    RPC/s
[W] Avg: 10637    RPC/s Min: 10637    RPC/s Max: 10637    RPC/s
[LNet Bandwidth of lfrom]
[R] Avg: 10639.50 MiB/s Min: 10639.50 MiB/s Max: 10639.50 MiB/s
[W] Avg: 1.62     MiB/s Min: 1.62     MiB/s Max: 1.62     MiB/s
[LNet Rates of lto]
[R] Avg: 10637    RPC/s Min: 10637    RPC/s Max: 10637    RPC/s
[W] Avg: 21276    RPC/s Min: 21276    RPC/s Max: 21276    RPC/s
[LNet Bandwidth of lto]
[R] Avg: 1.62     MiB/s Min: 1.62     MiB/s Max: 1.62     MiB/s
[W] Avg: 10639.50 MiB/s Min: 10639.50 MiB/s Max: 10639.50 MiB/s
[LNet Rates of lfrom]
[R] Avg: 21297    RPC/s Min: 21297    RPC/s Max: 21297    RPC/s
[W] Avg: 10649    RPC/s Min: 10649    RPC/s Max: 10649    RPC/s
[LNet Bandwidth of lfrom]
[R] Avg: 10649.22 MiB/s Min: 10649.22 MiB/s Max: 10649.22 MiB/s
[W] Avg: 1.62     MiB/s Min: 1.62     MiB/s Max: 1.62     MiB/s
[LNet Rates of lto]
[R] Avg: 10648    RPC/s Min: 10648    RPC/s Max: 10648    RPC/s
[W] Avg: 21294    RPC/s Min: 21294    RPC/s Max: 21294    RPC/s
[LNet Bandwidth of lto]
[R] Avg: 1.62     MiB/s Min: 1.62     MiB/s Max: 1.62     MiB/s
[W] Avg: 10649.22 MiB/s Min: 10649.22 MiB/s Max: 10649.22 MiB/s
[LNet Rates of lfrom]
[R] Avg: 21304    RPC/s Min: 21304    RPC/s Max: 21304    RPC/s
[W] Avg: 10652    RPC/s Min: 10652    RPC/s Max: 10652    RPC/s
[LNet Bandwidth of lfrom]
[R] Avg: 10653.70 MiB/s Min: 10653.70 MiB/s Max: 10653.70 MiB/s
[W] Avg: 1.63     MiB/s Min: 1.63     MiB/s Max: 1.63     MiB/s
[LNet Rates of lto]
[R] Avg: 10652    RPC/s Min: 10652    RPC/s Max: 10652    RPC/s
[W] Avg: 21305    RPC/s Min: 21305    RPC/s Max: 21305    RPC/s
[LNet Bandwidth of lto]
[R] Avg: 1.63     MiB/s Min: 1.63     MiB/s Max: 1.63     MiB/s
[W] Avg: 10653.70 MiB/s Min: 10653.70 MiB/s Max: 10653.70 MiB/s

lfrom:
Total 0 error nodes in lfrom
lto:
Total 0 error nodes in lto
1 batch in stopping
Batch is stopped
session is ended

The only issue I ran into was a panic trying to reboot. The servers required a powercyle.

Jul 12 19:09:14 ioperf-06 kernel: reboot          D    0 10092   9865 0x00004080
Jul 12 19:09:14 ioperf-06 kernel: Call Trace:
Jul 12 19:09:14 ioperf-06 kernel: __schedule+0x2c4/0x700
Jul 12 19:09:14 ioperf-06 kernel: ? __switch_to_asm+0x35/0x70
Jul 12 19:09:14 ioperf-06 kernel: ? __switch_to_asm+0x35/0x70
Jul 12 19:09:14 ioperf-06 kernel: schedule+0x38/0xa0
Jul 12 19:09:14 ioperf-06 kernel: schedule_timeout+0x246/0x2f0
Jul 12 19:09:14 ioperf-06 kernel: ? __switch_to_asm+0x41/0x70
Jul 12 19:09:14 ioperf-06 kernel: ? __switch_to+0x10c/0x480
Jul 12 19:09:14 ioperf-06 kernel: ? __schedule+0x2cc/0x700
Jul 12 19:09:14 ioperf-06 kernel: wait_for_completion+0x97/0x100
Jul 12 19:09:14 ioperf-06 kernel: cma_remove_one+0x23f/0x310 [rdma_cm]
Jul 12 19:09:14 ioperf-06 kernel: remove_client_context+0x8b/0xd0 [ib_core]
Jul 12 19:09:14 ioperf-06 kernel: disable_device+0x8c/0x130 [ib_core]
Jul 12 19:09:14 ioperf-06 kernel: __ib_unregister_device+0x35/0xa0 [ib_core]
Jul 12 19:09:14 ioperf-06 kernel: ib_unregister_device+0x21/0x30 [ib_core]
Jul 12 19:09:14 ioperf-06 kernel: __mlx5_ib_remove+0x38/0x60 [mlx5_ib]
Jul 12 19:09:14 ioperf-06 kernel: mlx5_detach_device+0xb2/0xc0 [mlx5_core]
Jul 12 19:09:14 ioperf-06 kernel: mlx5_unload_one+0x80/0x120 [mlx5_core]
Jul 12 19:09:14 ioperf-06 kernel: shutdown+0x144/0x1d0 [mlx5_core]
Jul 12 19:09:14 ioperf-06 kernel: pci_device_shutdown+0x34/0x60
Jul 12 19:09:14 ioperf-06 kernel: device_shutdown+0x161/0x212
Jul 12 19:09:14 ioperf-06 kernel: kernel_restart+0xe/0x30
Jul 12 19:09:14 ioperf-06 kernel: __do_sys_reboot+0x1d2/0x210
Jul 12 19:09:14 ioperf-06 kernel: ? syscall_trace_enter+0x1d3/0x2c0
Jul 12 19:09:14 ioperf-06 kernel: ? __audit_syscall_exit+0x249/0x2a0
Jul 12 19:09:14 ioperf-06 kernel: do_syscall_64+0x5b/0x1a0
Jul 12 19:09:14 ioperf-06 kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca
Jul 12 19:09:14 ioperf-06 kernel: RIP: 0033:0x7f73aa2825f7
Jul 12 19:09:14 ioperf-06 kernel: Code: Unable to access opcode bytes at RIP 0x7f73aa2825cd.
Jul 12 19:09:14 ioperf-06 kernel: RSP: 002b:00007ffed2874d28 EFLAGS: 00000246 ORIG_RAX: 00000000000000a9
Jul 12 19:09:14 ioperf-06 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f73aa2825f7
Jul 12 19:09:14 ioperf-06 kernel: RDX: 0000000001234567 RSI: 0000000028121969 RDI: 00000000fee1dead
Jul 12 19:09:14 ioperf-06 kernel: RBP: 00007ffed2874d70 R08: 0000000000000002 R09: 0000000000000000
Jul 12 19:09:14 ioperf-06 kernel: R10: 000000000000004b R11: 0000000000000246 R12: 0000000000000001
Jul 12 19:09:14 ioperf-06 kernel: R13: 00000000fffffffe R14: 0000000000000006 R15: 0000000000000000

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44217
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 8c70059776f2bdc7a68a27e35bb5dc36763d3dd6