[LU-9120] LNet Network Health Feature

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.12.0
• Type: New Feature
• Priority: Minor
• Reporter: Amir Shehata (Inactive)
• Assignee: Amir Shehata (Inactive)
• Resolution: Fixed

Description

LNet Multi-Rail has implemented the ability for multiple interfaces to be used on the same LNet network or across multiple LNet networks. The LNet Health feature adds the ability to resend messages across different interfaces when interface or network failures are detected. This allows LNet to mitigate communication failures before passing the failures to upper layers for further error handling. To accomplish this, LNet Health depends on health information reported by the underlying fabrics such as MLX and OPA.
LNet Health will monitor three different types of failures:

• local interface failures as reported by the underlying fabric
• remote interface failures as reported by the remote fabric
• network timeouts.
  Each one of these classes of failures are dealt with separately at the LNet layer. The implementation of this health feature at the LNet layer allows LNet to retransmit messages across different types of interfaces. For example if a peer has both MLX and OPA interfaces and a transmit error is detected on one of them then LNet can retransmit the message on the other available interface.

Issue Links

• is blocked by: LUDOC-396 Add documentation for the LNet Health...
• is related to: LU-11271 LNet Health: o2iblnd, conditionally s...
• is related to: LU-11272 LNet Health: handle routing special case
• is related to: LU-11422 Make LNet Selftest post Health backwa...
• is related to: LU-11273 LNet Health: update logging
• is related to: LU-7734 LNet Multi-Rail Project
• is related to: LU-10756 Send Uevents for interesting Lustre c...
• is related to: LU-13510 Allow control over LND timeouts indep...

Comments

[Colin Faber [X] (Inactive)]

So with this feature, does it provide functionality which isn't actually present in the current Multi-Rail feature?

[Joseph Gmitter (Inactive)]

Yes, the work being described here is a future addition to what was added with Multi-Rail in Lustre 2.10.X.

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32760
Subject: LU-9120 lnet: refactor lnet_select_pathway()
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 2d2769a6dd105d8e2993a5e4446dbd9a10a04d74

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32775
Subject: LU-9120 lnet: add health statistics
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 4e80bbb1753a607ceae3dc5f4d25f8acc457aa72

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32771
Subject: LU-9120 lnet: timeout delayed REPLYs and ACKs
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 0d7d91b6e4f2dc5243675942b9e5781b1bf012bd

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32769
Subject: LU-9120 lnet: add retry count
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 1c3293b585a8c49c3254c05f35a88dfedde1e3a6

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32773
Subject: LU-9120 lnet: reset health value
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: eaa6e62c51eccce8409f8f031c357537349199af

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32772
Subject: LU-9120 lnet: handle fatal device error
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 40b41527aeba77398019f9fd4793dae5a269e1e7

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32776
Subject: LU-9120 lnet: Add ioctl to get health stats
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 5bfdc16b84c084f308b45b5e2d52239a628b5e9d

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32770
Subject: LU-9120 lnet: calculate the lnd timeout
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 89260e1efc02b6a655e8696fe65308d39d9538dd

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32774
Subject: LU-9120 lnet: override global message timeout
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: e4fb080addd2ab46113a2ece9a6a844159f4a5b8

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32761
Subject: LU-9120 lnet: add health value per ni
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: a47a93879698bfc58356eb6f4a91aea42982ba42

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32765
Subject: LU-9120 lnet: handle o2iblnd tx failure
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: ae774b3ad57403a698f3e7ee3d995ee75e536fc8

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32766
Subject: LU-9120 lnet: handle socklnd tx failure
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: a236f8457113153779570274478f706b0b4b6b6b

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32762
Subject: LU-9120 lnet: add lnet_health_sensitivity
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 5840e32e2356fc88330dc9298cfbc3a0c095ae07

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32767
Subject: LU-9120 lnet: handle remote errors in LNet
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: a4cd0e553a7eedfb0a67121b0b50b36cf8bdd2b0

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32763
Subject: LU-9120 lnet: add monitor thread
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 2ae9c1bdae21ab7d59b54ecc9888e3fa986a2259

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32764
Subject: LU-9120 lnet: handle local ni failure
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: d02fa39356c02eb469333375dc152b40e3ee6c75

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32768
Subject: LU-9120 lnet: implement lnd_drop_msgs()
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: fcfda1d68e90a9b0ed2ad41159533ce3e58451f7

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32777
Subject: LU-9120 lnet: set retry count from lnetctl
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: cd0af48a5e42c1b7a0dea58d754780c9fbaf6277

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32778
Subject: LU-9120 lnet: set transaction timeout from lnetctl
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 74cef1aa4fa310ff372ee6e371efa9a757d42f56

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32779
Subject: LU-9120 lnet: set health sensitivity from lnetctl
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 587d8dc65d1f229e62fbb4ab499a3c83b8e81829

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32782
Subject: LU-9120 lnet: show local ni health stats
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: cca3e0c7fde3139c4a34c53f34ee6f20356a0d04

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32783
Subject: LU-9120 lnet: show peer ni health stats
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 1d0fdd44631fcacb7c24ace61fcffa319ca9520f

[Chris Horn]

Amir, would you add me as a reviewer to these patches when you are ready for reviews?

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32862
Subject: LU-9120 lnet: remove obsolete health functions
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 3a86c9944502bffc9ce145556802c9906b935422

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32861
Subject: LU-9120 lnet: sysfs functions for module params
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: dea54ccaea2ec2e2cbb8ae26cdb5e8fb1d04565f

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32863
Subject: LU-9120 lnet: set health value from user space
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 7fe60437d3657400677f10a4f69138eea372ace6

[Peter Jones]

hornc

The patches are ready for review now - https://review.whamcloud.com/#/q/status:open+project:fs/lustre-release+branch:multi-rail

Peter

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32950
Subject: LU-9120 lnet: print recovery queues content
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 76ef3b6ba1b00ea631f2d0e82b84ffad7871884f

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32949
Subject: LU-9120 lnet: keep track of resent messages
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: be3e441cb5876d90c83be7f0a7f1c2c4a7e16c2d

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32951
Subject: LU-9120 lnet: health error simulation
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 4de8ad54a5703b217035715c842b5682591cb70e

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32992
Subject: LU-9120 lnet: remove duplicate timeout mechanism
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set: 1
Commit: 3dbdebcfb26c2703e5f94e772a5d119c070bf7f2

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32760/
Subject: LU-9120 lnet: refactor lnet_select_pathway()
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 4e48761a57193279ce3f3d5170c3e38cf287b59a

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32761/
Subject: LU-9120 lnet: add health value per ni
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: d54afb86116c0640d7a201571b337042c87a3e40

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32762/
Subject: LU-9120 lnet: add lnet_health_sensitivity
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 63cf744d0fdf72fc5ac7e154ec60c4a08139acc4

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32763/
Subject: LU-9120 lnet: add monitor thread
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: b01e6fce1c988139b5fe59484c7568362992f37b

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32764/
Subject: LU-9120 lnet: handle local ni failure
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 70616605dd44be37068f4e1a4745a2f8b90eb1f5

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32765/
Subject: LU-9120 lnet: handle o2iblnd tx failure
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 8cf835e425d845da2ad3a787898a2b3001f75114

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32766/
Subject: LU-9120 lnet: handle socklnd tx failure
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 25c1cb2c4d6f4430c8e1be915f5e8742ba16a94c

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32767/
Subject: LU-9120 lnet: handle remote errors in LNet
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 76fad19c2deaa72b5b70eff4bf9d84e20a42a74e

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32769/
Subject: LU-9120 lnet: add retry count
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 20e23980eae2341c04688b6409442673516cb2c0

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32770/
Subject: LU-9120 lnet: calculate the lnd timeout
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 84f3af43c4bdeb1744736f44cd746dd4b6e8fa6d

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32861/
Subject: LU-9120 lnet: sysfs functions for module params
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 5169827bf79071d47d2b6d76e110fa412ff1fb38

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32771/
Subject: LU-9120 lnet: timeout delayed REPLYs and ACKs
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: a57fa1176e74ea549e0d63cd8753b97561dd8bbf

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32992/
Subject: LU-9120 lnet: remove duplicate timeout mechanism
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 0b1947d14188db34de0aa12b8d21e0b09e00ae13

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32772/
Subject: LU-9120 lnet: handle fatal device error
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 6b1571209a9938719b081465f1ee327380a70554

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32773/
Subject: LU-9120 lnet: reset health value
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 2f5a6d1233ac19fb46e9f9d18364e86eee0d6f19

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32775/
Subject: LU-9120 lnet: add health statistics
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 67908ab34371bdfe54b79323c7f570e3ce826170

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32776/
Subject: LU-9120 lnet: Add ioctl to get health stats
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 10958cac798db5c384f64ee1d48fc2b5ba423f0a

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32862/
Subject: LU-9120 lnet: remove obsolete health functions
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: ba05b3a98a0c41d31968b9c9e9ee66cbc68504e8

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32777/
Subject: LU-9120 lnet: set retry count from lnetctl
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 29f21e2c850a77676e44df0cea746bfa5c6f19ad

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32778/
Subject: LU-9120 lnet: set transaction timeout from lnetctl
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: cf47570b0273e11f81be0fc67126172fe73ad367

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32779/
Subject: LU-9120 lnet: set health sensitivity from lnetctl
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 4a7357d5e945e5fe89e03b5d65edb45af95b49ee

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32782/
Subject: LU-9120 lnet: show local ni health stats
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 60fc3c74757025f03ddd4b8e322716811ae97d3c

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32783/
Subject: LU-9120 lnet: show peer ni health stats
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: f64abb795a893d8208688350220c69e808d0af69

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32863/
Subject: LU-9120 lnet: set health value from user space
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: c0ad398fd71610c42b7ed06f8d2ca722daa01391

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32949/
Subject: LU-9120 lnet: add global health statistics
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 15020fd977af68620e862ad999eaab17688933e2

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32950/
Subject: LU-9120 lnet: print recovery queues content
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 826ea19c077b2a3e1a32464a7eb63fba6e460946

[Gerrit Updater]

Amir Shehata ([e-mail]) merged in patch https://review.whamcloud.com/32951/
Subject: LU-9120 lnet: health error simulation
Project: fs/lustre-release
Branch: multi-rail
Current Patch Set:
Commit: 5c17777d97bd20cde68771c6186320b5eae90e62

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33023
Subject: LU-9120 lnet: LNet Health/Resiliency Feature
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 958ef71f33fa925e6657f9902702cd3677e15ec9

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33037
Subject: LU-9120 lnet: LNet Health/Resiliency Feature
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3461795c2a407aa99b7e178c275367e27a3ddc42

[Peter Jones]

Feature landed for 2.12. Any bug fixes etc will be tracked under new tickets