[LU-15234] LNet high peer reference counts inconsistent with queue

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Labels: llnl
• Environment: lustre-2.12.7_2.llnl-2.ch6.x86_64 ; 3.10.0-1160.45.1.1chaos.ch6.x86_64
• Severity: 3
• Assignee: Serguei Smirnov
• Reporter: Olaf Faaland

Description

I believe that peer reference counts may not be decremented in some LNet error path, or that the size of the queue is not accurately reported by "lctl get_param peers".

The reference counts reported as "refs" by "lctl get_param peers" are increasing linearly with time. This is in contrast with "queue" which periodically spikes but then drops to 0 again.  Below shows 4 routers on ruby which have refs > 46,000 for a route to 72.19.2.24@o2ib100 even though the reported queue is 0.  This is just a little over 6 days since the ruby routers were rebooted during an update.

[root@ruby1009:~]# pdsh -v -g router lctl get_param peers 2>/dev/null | awk '$3 > 20 {print}' | sed 's/^.*://' | sort -V -u
 172.19.2.24@o2ib100      46957    up     5     8 -46945 -46945     8   -13 0
 172.19.2.24@o2ib100      47380    up     1     8 -47368 -47368     8   -23 0
 172.19.2.24@o2ib100      48449    up    15     8 -48437 -48437     8   -17 0
 172.19.2.24@o2ib100      49999    up     3     8 -49987 -49987     8    -7 0

The ruby routers  have an intermittent LNet communication problem (the fabric itself seems fine according to several tests, so the underlying issue is still under investigation).

Issue Links

• duplicates: LU-12739 Race with discovery thread completion and message queueing
• is related to: LU-15453 MDT shutdown hangs on mutex_lock, possibly cld_lock

Comments

[Peter Jones]

Yes I think that we can mark this ticket as a duplicate of LU-12739 once 48190 has been merged to b2_12. It should be included in the next b2_12-next batch we test

[Olaf Faaland]

As far as I'm concerned, this will be resolved when the patch lands to b2_12.  Do you agree?  If so, what is the plan for that?

thanks

[Olaf Faaland]

Hi Serguei,

2.12.9 + change 48190 appears to have resolved this issue on orelic, which has been a reliable reproducer.

Olaf

[Olaf Faaland]

Hi Serguei,

2.12.9 + change 48190 held up well overnight which is far beyond how long we've needed to wait for symptoms in the past. If you can get someone to perform a second review on the patch in gerrit that would be great.

I'll deploy more widely and update here early next week.

thanks,
Olaf

[Olaf Faaland]

Hi Serguei

I tested 2.12.9 + change 48190 today and results so far are promising. I'll test it further and post here in the next couple of days.

[Serguei Smirnov]

Hi Olaf,

I ported Chris's fix for LU-12739 to b2_12: https://review.whamcloud.com/#/c/48190/

Please give this patch a try. It is aiming to eliminate a race condition with effects potentially similar to what is seen in the coredump you provided.

Thanks,

Serguei.

[Serguei Smirnov]

Chris,

Yes indeed, it looks very much like LU-12739

I'll port these changes.

Thanks,

Serguei.

[Chris Horn]

Sounds like https://jira.whamcloud.com/browse/LU-12739 ?

[Serguei Smirnov]

Hi Olaf,

While examining the core I found that messages causing the delay are waiting to be sent: they are listed on lp_dc_pendq of the destination peer.

At the same time, the destination peer is not queued to be discovered, so it appears that there's no good reason for the messages to be delayed.

I pushed a test patch in order to rule out a race condition which somehow enables a thread to queue a message for a peer which is not (or no longer) going to be discovered. The new patch is going to attempt to recognize this situation on discovery completion, print an error and handle any messages which are still pending. This should help locate the race condition if it is actually occurring. If this is the only cause, with this patch we should see the error message "Peer X msg list not empty on disc comp" and no more refcount increase.

Otherwise, I'll have to look for other possible causes.

Thanks,

Serguei.

[Gerrit Updater]

"Serguei Smirnov <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48163
Subject: LU-15234 lnet: test for race when completing discovery
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 0eb36b2ace98b0c57595098a3a6d9f5de8e6045c

[Serguei Smirnov]

Hi Olaf,

I had too many distractions and haven't finished looking at the core yet.

Basically I believe what I see in the core so far does confirm the idea that messages are not getting finalized, but I still haven't understood why. In LNet layer the number of queued messages on the problem peer looks consistent with the high refcount, but I still need to dig more at the LND level and examine message queues there.

Thanks,

Serguei.