[LU-12886] A lot of LNetError: lnet_peer_ni_add_to_recoveryq_locked() messages

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Affects Version/s: Lustre 2.12.3
• Labels: llnl
• Environment: 2.12.3 RC1 (vanilla) on servers, CentOS 7.6, patched kernel; 2.12.0 + patches on clients
• Severity: 4
• Assignee: Amir Shehata (Inactive)
• Reporter: Stephane Thiell

Description

After upgrading our servers on Fir (Sherlock's /scratch) to Lustre 2.12.3 RC1, we are noticing a lot of these messages on all Lustre servers:

LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900

The NIDs reported are our Lustre routers, that are still running 2.12.0+patches (they are on the client clusters).

Attaching logs from all servers as lnet_recoveryq.log

This doesn't seem to have an impact on production and so far 2.12.3 RC1 has been just great for us (and we run it without additional patches now!). Thanks!

Stéphane

Issue Links

• is related to: LU-13071 LNet Health: reduce log severity

Comments

[Olaf Faaland]

Agreed

[Peter Jones]

The https://review.whamcloud.com/#/c/37718/ fix landed to b2_12 some weeks back so I think that it should be ok to close this ticket. Any objections?

[Olaf Faaland]

Hi Stephane,

They've pushed a backport for b2_12 to gerrit, and it's been reviewed and passed tests, just needs to land:

https://review.whamcloud.com/#/c/37718/

[Stephane Thiell]

Hello, we would also appreciate a backport of this patch (that just landed into master) to b2_12 as it is very verbose. Thanks!!

[Amir Shehata (Inactive)]

Hi Olaf,

It doesn't look like this patch made it in 2.12.4. It still hasn't landed. Needs an extra review:

https://review.whamcloud.com/#/c/37002/

The "inconsistent" message was reduced here:

f549927ea633b910a8c788fa970af742b3bf10c1 LU-11981 lnet: clean up error message

thanks

amir

[Olaf Faaland]

Hi Amir,

What 2.12.4 patch(es) reduce the severity of the "added to recovery queue" messages?

thanks