[LU-16149] LNet Discovery queue and deletion race

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.3
• Affects Version/s: Lustre 2.16.0, Lustre 2.15.2
• Severity: 3
• Assignee: Chris Horn
• Reporter: Chris Horn

Description

lnet_peer_deletion() can race with another thread calling
lnet_peer_queue_for_discovery.

Discovery thread:

• Calls lnet_peer_deletion():
• LNET_PEER_DISCOVERING bit is cleared from lnet_peer::lp_state
• releases lnet_peer::lp_lock

Another thread:

• Acquires lnet_net_lock/EX
• Calls lnet_peer_queue_for_discovery()
• Takes lnet_peer::lp_lock
• Sets LNET_PEER_DISCOVERING bit
• Releases lnet_peer::lp_lock
• Sees lnet_peer::lp_dc_list is not empty, so it does not add peer
  to dc request queue
• lnet_peer_queue_for_discovery() returns, lnet_net_lock/EX releases

Discovery thread:

• Acquires lnet_net_lock/EX
• Deletes peer from ln_dc_working list
• performs the peer deletion

At this point, the peer is not on any discovery list, and it has
LNET_PEER_DISCOVERING bit set. This peer is now stranded, and any
messages on the peer's lnet_peer::lp_dc_pendq are likewise stranded.

To solve this, modify lnet_peer_deletion() so that it waits to clear
the LNET_PEER_DISCOVERING bit until it has completed deleting the
peer and re-acquired the lnet_peer::lp_lock. This ensures we cannot
race with any other thread that may add the LNET_PEER_DISCOVERING bit
back to the peer.

Futhermore, do not bother deleting the peer from the ln_dc_working
list in lnet_peer_deletion(). This will be done by
lnet_peer_discovery_complete().

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49772/
Subject: LU-16149 lnet: Discovery queue and deletion race
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 7caade3078a168c3d39f6318c485490322604ab4

[Gerrit Updater]

"Serguei Smirnov <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49772
Subject: LU-16149 lnet: Discovery queue and deletion race
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: d464cf8747032b92d0d0daa7a9a2153a2b30b6d5

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48532/
Subject: LU-16149 lnet: Discovery queue and deletion race
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a966b624ac76e34e8ed28c6980c3f58cb441eeb0

[Gerrit Updater]

"Chris Horn <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48532
Subject: LU-16149 lnet: Discovery queue and deletion race
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 27a2481daf1026883f004109bd9a766cf5798161