[LU-10124] lnetctl: lnetctl import --add not importing peers correctly

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.10.0, Lustre 2.10.1, Lustre 2.11.0
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.7
• Type: Bug
• Priority: Minor
• Reporter: Malcolm Haak - NCI (Inactive)
• Assignee: Sonia Sharma (Inactive)
• Resolution: Fixed
• Labels: lnet, lnetctl

Environment

Centos 7.4

• Epic/Theme: lnet, lustre-2.10.1
• Severity: 3

Description

When importing a yaml config file for peers the import does not correctly set the Multi-Rail property when it is false.

An example:

peer:
    - primary nid: 10.112.1.60@o2ib8
      Multi-Rail: False
      peer ni:
        - nid: 10.112.1.60@o2ib8
          state: up

When imported it results in a running config of:

peer:
    - primary nid: 10.112.1.60@o2ib8
      Multi-Rail: True
      peer ni:
        - nid: 10.112.1.60@o2ib8
          state: up

For our config this isn't an issue yet, but as we will have a mix of multi-rail and non-multi-rail nodes this could be an issue moving forward.

Comments

[Malcolm Haak - NCI (Inactive)]

We have encountered a second issue when importing peers.

Two peers with a single primary nid were merged during import into one peer with two peer ni's

The YAML file being used was one that was exported from the lnet router it was imported on.

Not sure if it is a race condition when importing 4000+ peers or some other issue.

[James Nunez (Inactive)]

Sonia,
Would you please look into this issue?

Thank you.

[Gerrit Updater]

Sonia Sharma ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31138
Subject: LU-10124 lnet: Correctly add peer MR value while importing
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ddd0aedbe7a26053ae9c3da2c82c9e7a3e5a7c76

[Malcolm Haak - NCI (Inactive)]

This patch should fix up that one issue,

But the second issue of peer merging does not appear to be solved.

Did you want a second ticket for this issue?

I will get you some log lines from the affected nodes.

[Kim Sebo]

log line on lnet router is:

LNetError: 8507:0:(peer.c:806:lnet_add_peer_ni_to_prim_lpni()) Cannot add NID 10.9.60.1@o2ib3 owned by peer 10.9.60.1@o2ib3 to peer 10.9.12.38@o2ib3

The two 10.9.x.x addresses mentioned correspond to adjacent entries in the config file.

[Sonia Sharma (Inactive)]

Is the issue happening even after applying the patch?
When the MR value is correctly imported, it would know that the peer is MR and thus another NID should be merged to the same peer.

[Malcolm Haak - NCI (Inactive)]

I think you misunderstand. It's merging peers that AREN'T supposed to be merged.

Say peer A is in the file with a nid of 10.9.12.38@o2ib3:

peer:
    - primary nid: 10.9.12.38@o2ib3
      Multi-Rail: True
      peer ni:
        - nid: 10.9.12.28@o2ib3
          state: up

and peer B is next in the YAML file with a nid of 10.9.60.1@o2ib3

peer:
    - primary nid: 10.9.60.1@o2ib3
      Multi-Rail: False
      peer ni:
        - nid: 10.9.60.1@o2ib3
          state: up

So the resulting peer config YAML file should look like

peer:
    - primary nid: 10.9.12.38@o2ib3
      Multi-Rail: False
      peer ni:
        - nid: 10.9.12.38@o2ib3
          state: up
    - primary nid: 10.9.60.1@o2ib3
      Multi-Rail: False
      peer ni:
        - nid: 10.9.60.1@o2ib3
          state: up

It's trying to add 10.9.60.1@o2ib3 as an extra peer ni to 10.9.12.38@o2ib3.

This is wrong. They are separate peers. I've checked the YAML file and they are both described in YAML correctly. There is something wrong with the YAML parser that it causing it to not parse correctly.

[Sonia Sharma (Inactive)]

Oh okay. So I noticed that I never updated the patch which had issue. Just did that.

And now with the patch, I just tried it on my system and I could not replicate the issue.

[root@lutfRtr1-linux ~]# lnetctl ping 10.211.55.9@tcp
ping:
    - primary nid: 10.211.55.9@tcp
      Multi-Rail: False
      peer ni:
        - nid: 10.211.55.9@tcp

[root@lutfRtr1-linux lustre-release]# lnetctl peer add --prim_nid 10.9.60.24@tcp

[root@lutfRtr1-linux lustre-release]# lnetctl peer show
peer:
    - primary nid: 10.211.55.9@tcp
      Multi-Rail: False
      peer ni:
        - nid: 10.211.55.9@tcp
          state: NA
    - primary nid: 10.9.60.24@tcp
      Multi-Rail: True
      peer ni:
        - nid: 10.9.60.24@tcp
          state: NA

[root@lutfRtr1-linux lustre-release]# lnetctl export > out.yaml

[root@lutfRtr1-linux lustre-release]# lnetctl peer show
peer:

[root@lutfRtr1-linux lustre-release]# lnetctl import < out.yaml

[root@lutfRtr1-linux lustre-release]# lnetctl peer show
peer:
    - primary nid: 10.211.55.9@tcp
      Multi-Rail: False
      peer ni:
        - nid: 10.211.55.9@tcp
          state: NA
    - primary nid: 10.9.60.24@tcp
      Multi-Rail: True
      peer ni:
        - nid: 10.9.60.24@tcp
          state: NA

How are you adding peers? Can you list the commands you are running to add peers.

Though I tried both ways - using "lnetctl" command and running traffic and was able to import peers correctly.

[Malcolm Haak - NCI (Inactive)]

I think its a race condition. It happens because we are importing ~4000 nodes while in production (so along side normal discovery)

I doubt you will trigger it with two.

[Sonia Sharma (Inactive)]

Hi Malcolm,

Can you please attach here the YAML file you are using for configuration. We can try reproducing the issue using that YAML file.

Thanks

[Gerrit Updater]

Sonia Sharma ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32255
Subject: LU-10124 lnet: Correctly add peer MR value while importing
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: c6fcf5a01fa4da0b026498b16927fa6c86cc1918

[Sonia Sharma (Inactive)]

Just pushed the back-ported patch for b2_10 to make it easy for you to apply the patch and test.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31138/
Subject: LU-10124 lnet: Correctly add peer MR value while importing
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 99494a28e6afde7c59e7f03045e63028ece1064d

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32255/
Subject: LU-10124 lnet: Correctly add peer MR value while importing
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 8103e94c1bd3000bc25da0d05f0ef3cafa1f91fd