[LU-16709] LNet: locking multiple NIDs of the same MR peer as primary results in incorrect representation

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.5
• Labels: lnet multi-rail
• Severity: 3
• Assignee: Serguei Smirnov
• Reporter: Serguei Smirnov

Description

System configurations which result in Lustre layer specifying the same MR peer using multiple NIDs cause an issue with primary NID locking logic: when "primary nid locking" feature is enabled, LNet creates separate peer records, each record containing one NID of the MR peer as "locked primary". After the discovery completes in the background, these records are not being merged. This results in incorrect peer representation. Here's an example:

server:

# lnetctl net show
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
    - net type: tcp
      local NI(s):
        - nid: 192.168.122.50@tcp
          status: up
          interfaces:
              0: eth0
        - nid: 192.168.122.134@tcp
          status: up
          interfaces:
              0: ens12

client:

# mount -t lustre 192.168.122.134@tcp:192.168.122.50@tcp:/lustrewt /mnt/lustrefs
# lnetctl peer show 
peer:
    - primary nid: 192.168.122.134@tcp
      Multi-Rail: True
      peer ni:
        - nid: 192.168.122.134@tcp
          state: NA
    - primary nid: 192.168.122.50@tcp
      Multi-Rail: True
      peer ni:
        - nid: 192.168.122.50@tcp
          state: NA

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/55486/
Subject: LU-16709 lnet: fix locking multiple NIDs of the MR peer
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 55f82ea129035cd64832be11f24821fd5f1b80f9

[Peter Jones]

Thanks Chris!

[Gerrit Updater]

"Frederick Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55486
Subject: LU-16709 lnet: fix locking multiple NIDs of the MR peer
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 79e4133f30e924ad552e9e8a53662c65c14476f6

[Serguei Smirnov]

hornc, yes I agree this patch should be backported too. I'll take care of this.

[Chris Horn]

ssmirnov pjones is this patch needed in b2_15 since the LU-14668 patch series was backported there?

[Peter Jones]

Seems to have merged for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50530/
Subject: LU-16709 lnet: fix locking multiple NIDs of the MR peer
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3b7a02ee4d656b7b3e044713681da2f56dddb152

[Serguei Smirnov]

Hi Chris,

For example, this sequence also leads to the similar "broken" peer record:

1. bring down link corresponding to 192.168.122.134@tcp on the server
2. execute "mount -t lustre 192.168.122.134@tcp:/lustrewt /mnt/lustrefs on the client" - it fails as expected, peer containing 192.168.122.134@tcp is created
3. execute "lnetctl discover 192.168.122.50@tcp on the client" - this succeeds
4. lnetctl peer show outputs separate peer records for 192.168.122.134@tcp and 192.168.122.50@tcp

Not sure if this reproducer is valid either though because mount command is not listing both server NIDs. 

[Shuichi Ihara]

In my case, the problem still exists even server and server connections without client mount below.

master without patch

# clush -a lnetctl peer show | dshbak  
----------------
ai400x2-3-vm1
----------------
peer:
    - primary nid: 10.0.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.202@o2ib12
          state: NA
        - nid: 10.1.11.202@o2ib12
          state: NA
    - primary nid: 10.0.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.203@o2ib12
          state: NA
        - nid: 10.1.11.203@o2ib12
          state: NA
    - primary nid: 0@lo
      Multi-Rail: False
      peer ni:
        - nid: 0@lo
          state: NA
    - primary nid: 10.0.11.200@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.200@o2ib12
          state: NA
        - nid: 10.1.11.200@o2ib12
          state: NA
    - primary nid: 10.0.11.201@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.201@o2ib12
          state: NA
        - nid: 10.1.11.201@o2ib12
          state: NA
----------------
ai400x2-3-vm2
----------------
peer:
    - primary nid: 10.0.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.202@o2ib12
          state: NA
    - primary nid: 10.1.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.1.11.202@o2ib12
          state: NA
    - primary nid: 10.0.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.203@o2ib12
          state: NA
    - primary nid: 10.1.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.1.11.203@o2ib12
          state: NA
    - primary nid: 0@lo
      Multi-Rail: False
      peer ni:
        - nid: 0@lo
          state: NA
    - primary nid: 10.0.11.200@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.200@o2ib12
          state: NA
        - nid: 10.1.11.200@o2ib12
          state: NA
    - primary nid: 10.0.11.201@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.201@o2ib12
          state: NA
        - nid: 10.1.11.201@o2ib12
          state: NA
----------------
ai400x2-3-vm3
----------------
peer:
    - primary nid: 10.0.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.202@o2ib12
          state: NA
    - primary nid: 10.1.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.1.11.202@o2ib12
          state: NA
    - primary nid: 10.0.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.203@o2ib12
          state: NA
    - primary nid: 10.1.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.1.11.203@o2ib12
          state: NA
    - primary nid: 0@lo
      Multi-Rail: False
      peer ni:
        - nid: 0@lo
          state: NA
    - primary nid: 10.0.11.200@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.200@o2ib12
          state: NA
        - nid: 10.1.11.200@o2ib12
          state: NA
    - primary nid: 10.0.11.201@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.201@o2ib12
          state: NA
        - nid: 10.1.11.201@o2ib12
          state: NA
----------------
ai400x2-3-vm4
----------------
peer:
    - primary nid: 10.0.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.202@o2ib12
          state: NA
        - nid: 10.1.11.202@o2ib12
          state: NA
    - primary nid: 10.0.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.203@o2ib12
          state: NA
    - primary nid: 10.1.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.1.11.203@o2ib12
          state: NA
    - primary nid: 0@lo
      Multi-Rail: False
      peer ni:
        - nid: 0@lo
          state: NA
    - primary nid: 10.0.11.200@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.200@o2ib12
          state: NA
        - nid: 10.1.11.200@o2ib12
          state: NA
    - primary nid: 10.0.11.201@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.201@o2ib12
          state: NA
        - nid: 10.1.11.201@o2ib12
          state: NA

After patch https://review.whamcloud.com/#/c/fs/lustre-release/+/50530/
the problem resolved and all MR state shows properly.

# clush -a lnetctl peer show | dshbak 
----------------
ai400x2-3-vm1
----------------
peer:
    - primary nid: 10.0.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.202@o2ib12
          state: NA
        - nid: 10.1.11.202@o2ib12
          state: NA
    - primary nid: 10.0.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.203@o2ib12
          state: NA
        - nid: 10.1.11.203@o2ib12
          state: NA
    - primary nid: 0@lo
      Multi-Rail: False
      peer ni:
        - nid: 0@lo
          state: NA
    - primary nid: 10.0.11.200@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.200@o2ib12
          state: NA
        - nid: 10.1.11.200@o2ib12
          state: NA
    - primary nid: 10.0.11.201@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.201@o2ib12
          state: NA
        - nid: 10.1.11.201@o2ib12
          state: NA
----------------
ai400x2-3-vm2
----------------
peer:
    - primary nid: 10.0.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.202@o2ib12
          state: NA
        - nid: 10.1.11.202@o2ib12
          state: NA
    - primary nid: 10.0.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.203@o2ib12
          state: NA
        - nid: 10.1.11.203@o2ib12
          state: NA
    - primary nid: 0@lo
      Multi-Rail: False
      peer ni:
        - nid: 0@lo
          state: NA
    - primary nid: 10.0.11.200@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.200@o2ib12
          state: NA
        - nid: 10.1.11.200@o2ib12
          state: NA
    - primary nid: 10.0.11.201@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.201@o2ib12
          state: NA
        - nid: 10.1.11.201@o2ib12
          state: NA
----------------
ai400x2-3-vm3
----------------
peer:
    - primary nid: 10.0.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.202@o2ib12
          state: NA
        - nid: 10.1.11.202@o2ib12
          state: NA
    - primary nid: 10.0.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.203@o2ib12
          state: NA
        - nid: 10.1.11.203@o2ib12
          state: NA
    - primary nid: 0@lo
      Multi-Rail: False
      peer ni:
        - nid: 0@lo
          state: NA
    - primary nid: 10.0.11.200@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.200@o2ib12
          state: NA
        - nid: 10.1.11.200@o2ib12
          state: NA
    - primary nid: 10.0.11.201@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.201@o2ib12
          state: NA
        - nid: 10.1.11.201@o2ib12
          state: NA
----------------
ai400x2-3-vm4
----------------
peer:
    - primary nid: 10.0.11.202@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.202@o2ib12
          state: NA
        - nid: 10.1.11.202@o2ib12
          state: NA
    - primary nid: 10.0.11.203@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.203@o2ib12
          state: NA
        - nid: 10.1.11.203@o2ib12
          state: NA
    - primary nid: 0@lo
      Multi-Rail: False
      peer ni:
        - nid: 0@lo
          state: NA
    - primary nid: 10.0.11.200@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.200@o2ib12
          state: NA
        - nid: 10.1.11.200@o2ib12
          state: NA
    - primary nid: 10.0.11.201@o2ib12
      Multi-Rail: True
      peer ni:
        - nid: 10.0.11.201@o2ib12
          state: NA
        - nid: 10.1.11.201@o2ib12
          state: NA

I also attached llog dump for a reference.

[Chris Horn]

# mount -t lustre 192.168.122.134@tcp:192.168.122.50@tcp:/lustrewt /mnt/lustrefs

This is an incorrect NID specification, isn't it? NIDs belonging to the same server are supposed to be comma-separated. NIDs belonging to other servers should be colon-separated.

LNet arguably did the correct thing here.