https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14499 Lustre <p>Changes applied by the patches from <a href="https://jira.whamcloud.com/browse/LU-13368" title="lnet may be trying to use deleted routes leading to errors kiblnd_rejected(): 10.0.10.212@o2ib7 rejected: consumer defined fatal error" class="issue-link" data-issue-key="LU-13368"><del>LU-13368</del></a> appear to be causing the o2iblnd shutdown procedure to not complete properly sometimes on lustre_rmmod:</p> <p>In that case, messages similar to the following keep showing up in the log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [51025.354675] LNet: 9402:0:(o2iblnd.c:3107:kiblnd_shutdown()) 10.1.11.124@o2ib10: waiting <span class="code-keyword">for</span> 3 peers to disconnect [51029.354481] LNet: 9402:0:(o2iblnd.c:3107:kiblnd_shutdown()) 10.1.11.124@o2ib10: waiting <span class="code-keyword">for</span> 3 peers to disconnect [51037.353971] LNet: 9402:0:(o2iblnd.c:3107:kiblnd_shutdown()) 10.1.11.124@o2ib10: waiting <span class="code-keyword">for</span> 3 peers to disconnect </pre> </div></div> <p> </p> LU-14499

o2iblnd: LU-13368 changes cause shutdown procedure to not complete

Bug Major Open Unresolved Serguei Smirnov Serguei Smirnov Mon, 8 Mar 2021 17:37:11 +0000 Mon, 30 Jan 2023 17:25:46 +0000 0 8 <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41937" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41937</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14499" title="o2iblnd: LU-13368 changes cause shutdown procedure to not complete" class="issue-link" data-issue-key="LU-14499">LU-14499</a> lnet: Revert "<a href="https://jira.whamcloud.com/browse/LU-13368" title="lnet may be trying to use deleted routes leading to errors kiblnd_rejected(): 10.0.10.212@o2ib7 rejected: consumer defined fatal error" class="issue-link" data-issue-key="LU-13368"><del>LU-13368</del></a> lnet: discard the callback"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: eda619ef352141de76b3bc2fe97d56ed68c7c9d9</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ssmirnov" class="user-hover" rel="ssmirnov">ssmirnov</a> could this issue impact ksocklnd as well?</p> <p>Chris,</p> <p>Despite concluding that <a href="https://jira.whamcloud.com/browse/LU-13368" title="lnet may be trying to use deleted routes leading to errors kiblnd_rejected(): 10.0.10.212@o2ib7 rejected: consumer defined fatal error" class="issue-link" data-issue-key="LU-13368"><del>LU-13368</del></a> patch is causing the issue, I never had complete understanding of what was going wrong exactly. It could be specific to o2iblnd only. I don't think I recall socklnd getting stuck in the same manner. </p> <p>We traced a memory leak back to the <a href="https://jira.whamcloud.com/browse/LU-13368" title="lnet may be trying to use deleted routes leading to errors kiblnd_rejected(): 10.0.10.212@o2ib7 rejected: consumer defined fatal error" class="issue-link" data-issue-key="LU-13368"><del>LU-13368</del></a> change. Given Alexey's prior misgivings about the patch, and its known bugginess, I think we should revert it.</p> <p>Hi Serguei,<br/> Is this stuck because you need more information?<br/> thanks,</p> <p>Hi Olaf,</p> <p>From comments in <a href="https://jira.whamcloud.com/browse/LU-13368" title="lnet may be trying to use deleted routes leading to errors kiblnd_rejected(): 10.0.10.212@o2ib7 rejected: consumer defined fatal error" class="issue-link" data-issue-key="LU-13368"><del>LU-13368</del></a> and this ticket, it looks like "lnet: discard the callback" change should be reverted. On the other hand, there were potential fixes supplied by Yang Sheng which didn't get tested. If I remember correctly, this got stuck pending the test results, which would help decide whether to revert the change, or keep it and add the fixes. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ys" class="user-hover" rel="ys">ys</a>, <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sihara" class="user-hover" rel="sihara">sihara</a>: is my understanding correct? </p> <p>Thanks,</p> <p>Serguei. </p> <p>Hi, Serguei, Yes, you are right. </p> <p>What are the gerrit URLs for those changes? Thanks.</p> <p>Hi Olaf,</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ys" class="user-hover" rel="ys">ys</a> will correct me if I'm wrong, but I believe these are the two changes which are supposed to be fixing the original "discard the callback":</p> <p><a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/40937/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/40937/</a></p> <p><a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/41970/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/41970/</a></p> <p>Thanks,</p> <p>Serguei.</p> <p> </p> <p>Sorry for the delay. Yes, Serguei is right. <br/> The <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/38845/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/38845/</a> is original patch. <br/> The <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/40937/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/40937/</a> is a patch to work with 38845 to provide full function.<br/> The <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/41970/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/41970/</a> is a bug fixing patch for this ticket. Since i think it should be tested first, So i mark it as a 'test patch'. </p> <p>Hi Serguei and Yang Sheng,</p> <p>Thanks for clarifying. It looks like changes 40937 and 41970 aren't progressing. Are you waiting on something?</p> <p>Thanks</p> <p>Hi Olaf, the problem here appears to be that even though the patches are code-complete and Maloo-tested, we're not able to verify Yang Sheng's fixes in a proper IB environment as Shuichi doesn't have the available resources. Would you be able to give these patches a try on your system?</p> <p> </p> Related LU-13638 LU-13368 Development Rank 1|i01otj: Rank (Obsolete) 9223372036854775807 Severity