[LU-14499] o2iblnd: LU-13368 changes cause shutdown procedure to not complete

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0
• Severity: 3
• Assignee: Serguei Smirnov
• Reporter: Serguei Smirnov

Description

Changes applied by the patches from LU-13368 appear to be causing the o2iblnd shutdown procedure to not complete properly sometimes on lustre_rmmod:

In that case, messages similar to the following keep showing up in the log:

[51025.354675] LNet: 9402:0:(o2iblnd.c:3107:kiblnd_shutdown()) 10.1.11.124@o2ib10: waiting for 3 peers to disconnect
[51029.354481] LNet: 9402:0:(o2iblnd.c:3107:kiblnd_shutdown()) 10.1.11.124@o2ib10: waiting for 3 peers to disconnect
[51037.353971] LNet: 9402:0:(o2iblnd.c:3107:kiblnd_shutdown()) 10.1.11.124@o2ib10: waiting for 3 peers to disconnect

 

Issue Links

• is related to: LU-13368 lnet may be trying to use deleted routes leading to errors kiblnd_rejected(): 10.0.10.212@o2ib7 rejected: consumer defined fatal error
• is related to: LU-13638 lnet: discard the callback

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/41937/
Subject: LU-14499 lnet: Revert "LU-13368 lnet: discard the callback"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fc28666b2f648dc3d52f7ceaeb552405e17883da

[Serguei Smirnov]

Hi Olaf, the problem here appears to be that even though the patches are code-complete and Maloo-tested, we're not able to verify Yang Sheng's fixes in a proper IB environment as Shuichi doesn't have the available resources. Would you be able to give these patches a try on your system?

 

[Olaf Faaland]

Hi Serguei and Yang Sheng,

Thanks for clarifying. It looks like changes 40937 and 41970 aren't progressing. Are you waiting on something?

Thanks

[Yang Sheng]

Sorry for the delay. Yes, Serguei is right.
The https://review.whamcloud.com/#/c/fs/lustre-release/+/38845/ is original patch.
The https://review.whamcloud.com/#/c/fs/lustre-release/+/40937/ is a patch to work with 38845 to provide full function.
The https://review.whamcloud.com/#/c/fs/lustre-release/+/41970/ is a bug fixing patch for this ticket. Since i think it should be tested first, So i mark it as a 'test patch'.

[Serguei Smirnov]

Hi Olaf,

ys will correct me if I'm wrong, but I believe these are the two changes which are supposed to be fixing the original "discard the callback":

https://review.whamcloud.com/#/c/fs/lustre-release/+/40937/

https://review.whamcloud.com/#/c/fs/lustre-release/+/41970/

Thanks,

Serguei.

 

[Olaf Faaland]

What are the gerrit URLs for those changes? Thanks.

[Yang Sheng]

Hi, Serguei, Yes, you are right.

[Serguei Smirnov]

Hi Olaf,

From comments in LU-13368 and this ticket, it looks like "lnet: discard the callback" change should be reverted. On the other hand, there were potential fixes supplied by Yang Sheng which didn't get tested. If I remember correctly, this got stuck pending the test results, which would help decide whether to revert the change, or keep it and add the fixes. 

ys, sihara: is my understanding correct? 

Thanks,

Serguei. 

[Olaf Faaland]

Hi Serguei,
Is this stuck because you need more information?
thanks,

[Chris Horn]

We traced a memory leak back to the LU-13368 change. Given Alexey's prior misgivings about the patch, and its known bugginess, I think we should revert it.

[Serguei Smirnov]

Chris,

Despite concluding that LU-13368 patch is causing the issue, I never had complete understanding of what was going wrong exactly. It could be specific to o2iblnd only. I don't think I recall socklnd getting stuck in the same manner.