[LU-7676] OSS Servers stuck in connecting/disconnect loop

• Type: Bug
• Resolution: Duplicate
• Priority: Minor
• Severity: 1
• Assignee: Oleg Drokin
• Reporter: Mahmoud Hanafi

Description

We have had several OSS started to get into a state of disconnect and reconnect with clients. Sometimes they clear-up and then re-enter the same state later. Even with reboot the will enter into the same state.

Attaching Lustre Debug dump. Please advice on what additional info is need for debugging.

Issue Links

• is related to: LU-7054 ib_cm scalling issue when lustre clients connect to OSS
• is related to: LU-7569 IB leaf switch caused LNet routers to crash

Comments

[Peter Jones]

Actually fix landed under LU-7569

[Mahmoud Hanafi]

need to ad NASA label

[Jay Lan (Inactive)]

We need a b2_7_fe back port also. ATM we plan to stop running 2.7.1 until we receive the back port.

[Jay Lan (Inactive)]

It looks like http://review.whamcloud.com/#/c/18025/ is a backport of patch LU-7569 http://review.whamcloud.com/#/c/17892/ that I have been asking for. Thanks.

[Jay Lan (Inactive)]

Patch 18026 missed a newline at line 1909.

[Doug Oucharek (Inactive)]

NASA: Please apply both patches being discussed here: http://review.whamcloud.com/#/c/18025/ and http://review.whamcloud.com/18026.

[Doug Oucharek (Inactive)]

Liang: is your patch in addition to the one Amir ported or is it a replacement for it?

[Liang Zhen (Inactive)]

I checked my original patch, seems I forgot to call set_current_state() before schedule_timeout(), which can't really help because current thread wouldn't sleep. I have updated the patch uploaded by Amir (http://review.whamcloud.com/#/c/16470/), I also ported it to 2_5_fe (http://review.whamcloud.com/18026)

[Amir Shehata (Inactive)]

Ported the patch here:
http://review.whamcloud.com/#/c/18025/

[Amir Shehata (Inactive)]

I looked through the log file attached and I see 442 instances of connection races, which occurs when two nodes are attempting to reconnect. This could result in a flurry of reconnects, which could consume memory. There is a prototype patch that has been done to address the same issue on another site. I'm in the process of porting it to NASA's branch and I'll push it in later today for you to try.