[LU-15234] LNet high peer reference counts inconsistent with queue

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Labels: llnl
• Environment: lustre-2.12.7_2.llnl-2.ch6.x86_64 ; 3.10.0-1160.45.1.1chaos.ch6.x86_64
• Severity: 3
• Assignee: Serguei Smirnov
• Reporter: Olaf Faaland

Description

I believe that peer reference counts may not be decremented in some LNet error path, or that the size of the queue is not accurately reported by "lctl get_param peers".

The reference counts reported as "refs" by "lctl get_param peers" are increasing linearly with time. This is in contrast with "queue" which periodically spikes but then drops to 0 again.  Below shows 4 routers on ruby which have refs > 46,000 for a route to 72.19.2.24@o2ib100 even though the reported queue is 0.  This is just a little over 6 days since the ruby routers were rebooted during an update.

[root@ruby1009:~]# pdsh -v -g router lctl get_param peers 2>/dev/null | awk '$3 > 20 {print}' | sed 's/^.*://' | sort -V -u
 172.19.2.24@o2ib100      46957    up     5     8 -46945 -46945     8   -13 0
 172.19.2.24@o2ib100      47380    up     1     8 -47368 -47368     8   -23 0
 172.19.2.24@o2ib100      48449    up    15     8 -48437 -48437     8   -17 0
 172.19.2.24@o2ib100      49999    up     3     8 -49987 -49987     8    -7 0

The ruby routers  have an intermittent LNet communication problem (the fabric itself seems fine according to several tests, so the underlying issue is still under investigation).

Issue Links

• duplicates: LU-12739 Race with discovery thread completion and message queueing
• is related to: LU-15453 MDT shutdown hangs on mutex_lock, possibly cld_lock

Comments

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48566/
Subject: LU-15234 lnet: add mechanism for dumping lnd peer debug info
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 950e59ced18d49e9fdd31c1e9de43b89a0bc1c1d

[Serguei Smirnov]

No I would prefer to address the comments and land this patch. Even though it is not fixing anything for this ticket (it is a debugging enhancement), it happens to have been created as a result of investigating this issue. 

[Peter Jones]

I think it is really a call for ssmirnov . Do you still think that there is value in landing https://review.whamcloud.com/#/c/fs/lustre-release/+/48566/ or do you intend to abandon it in light of the review comments?

[Olaf Faaland]

>  The LU-12739 fix has landed to b2_12 but perhaps this ticket should remain open to track the landing of https://review.whamcloud.com/#/c/fs/lustre-release/+/48566/ ?

No opinion from me.

Thanks for getting this fixed.

[Peter Jones]

The LU-12739 fix has landed to b2_12 but perhaps this ticket should remain open to track the landing of https://review.whamcloud.com/#/c/fs/lustre-release/+/48566/?

[Gerrit Updater]

"Serguei Smirnov <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48566
Subject: LU-15234 lnet: add mechanism for dumping lnd peer debug info
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: dc704df0be48fc9f933e6f2c6fede3c5991a951a

[Peter Jones]

Yes I think that we can mark this ticket as a duplicate of LU-12739 once 48190 has been merged to b2_12. It should be included in the next b2_12-next batch we test

[Olaf Faaland]

As far as I'm concerned, this will be resolved when the patch lands to b2_12.  Do you agree?  If so, what is the plan for that?

thanks

[Olaf Faaland]

Hi Serguei,

2.12.9 + change 48190 appears to have resolved this issue on orelic, which has been a reliable reproducer.

Olaf