Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-1252] Imperative recovery bugs go here https://jira.whamcloud.com/browse/LU-1252 Lustre <p>contain for imperative recovery.</p> LU-1252

Imperative recovery bugs go here

Bug Minor Resolved Fixed Jinshan Xiong Jinshan Xiong Thu, 22 Mar 2012 18:50:48 +0000 Mon, 29 May 2017 04:14:30 +0000 Mon, 29 May 2017 04:14:30 +0000 0 6 <p>First patch is here <a href="http://review.whamcloud.com/#change,2371" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2371</a></p> <p>While testing with the patch I seen two bugs. One was for a bogus recovery time out as seen below. I don't think the clients are in recovery for 18446744073709551615s</p> <p>LustreError: 4486:0:(ldlm_lib.c:941:target_handle_connect()) lustre-OST000c: denying connection for new client 12@gni (fde45892-b2d3-a6d0-0ff6-b0e9b7d0740b): 18 clients in recovery for 18446744073709551615s</p> <p>The second is this report:</p> <p>[ 945.256721] Lustre: lustre-OST0004: Recovery over after 0:01, of 24 clients 23 recovered and 1 was evicted.</p> <p>Judging by the timestamps it took longer then 1 second to recovery.</p> <p>Hi James, can you please apply this patch: <a href="http://review.whamcloud.com/#change,1797" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1797</a></p> <p>when you do IR test next time. I found this patch helped a lot to reduce reconnecting time.</p> <p>Merged it to our build system. Will test tomorrow.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/323/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el6,inkernel #323</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 6026e9c13b2ce73a3e55eeced7407329ac00bce0)</p> <p> Result = SUCCESS<br/> tappro : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=6026e9c13b2ce73a3e55eeced7407329ac00bce0" class="external-link" target="_blank" rel="nofollow noopener">6026e9c13b2ce73a3e55eeced7407329ac00bce0</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_nids.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/323/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el5,inkernel #323</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 6026e9c13b2ce73a3e55eeced7407329ac00bce0)</p> <p> Result = SUCCESS<br/> tappro : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=6026e9c13b2ce73a3e55eeced7407329ac00bce0" class="external-link" target="_blank" rel="nofollow noopener">6026e9c13b2ce73a3e55eeced7407329ac00bce0</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_nids.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/323/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,server,el5,inkernel #323</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 6026e9c13b2ce73a3e55eeced7407329ac00bce0)</p> <p> Result = SUCCESS<br/> tappro : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=6026e9c13b2ce73a3e55eeced7407329ac00bce0" class="external-link" target="_blank" rel="nofollow noopener">6026e9c13b2ce73a3e55eeced7407329ac00bce0</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/mgs/mgs_nids.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/323/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el6,inkernel #323</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 6026e9c13b2ce73a3e55eeced7407329ac00bce0)</p> <p> Result = SUCCESS<br/> tappro : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=6026e9c13b2ce73a3e55eeced7407329ac00bce0" class="external-link" target="_blank" rel="nofollow noopener">6026e9c13b2ce73a3e55eeced7407329ac00bce0</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/mgs/mgs_nids.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/323/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el5,inkernel #323</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 6026e9c13b2ce73a3e55eeced7407329ac00bce0)</p> <p> Result = SUCCESS<br/> tappro : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=6026e9c13b2ce73a3e55eeced7407329ac00bce0" class="external-link" target="_blank" rel="nofollow noopener">6026e9c13b2ce73a3e55eeced7407329ac00bce0</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_nids.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/323/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el5,inkernel #323</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 6026e9c13b2ce73a3e55eeced7407329ac00bce0)</p> <p> Result = SUCCESS<br/> tappro : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=6026e9c13b2ce73a3e55eeced7407329ac00bce0" class="external-link" target="_blank" rel="nofollow noopener">6026e9c13b2ce73a3e55eeced7407329ac00bce0</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_nids.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/323/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el6,inkernel #323</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 6026e9c13b2ce73a3e55eeced7407329ac00bce0)</p> <p> Result = SUCCESS<br/> tappro : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=6026e9c13b2ce73a3e55eeced7407329ac00bce0" class="external-link" target="_blank" rel="nofollow noopener">6026e9c13b2ce73a3e55eeced7407329ac00bce0</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_nids.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Another patch is at: <a href="http://review.whamcloud.com/#change,2410" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2410</a></p> <p>Is this needed for my testings?</p> <p>It's only needed if you cluster is composed of heterogeneous nodes. So I don;t think you need apply it.</p> <p>I just got over 100GB of logs to look at. Its for one test run. In the test I attempted to powerman one OSS node. Well tanks to memory bugs in the the debug daemon all the OSS server went pop. The logs cover the entire length of recover. Server side I have systems log pre and post crash.</p> <p>It'll be really fun to look at 100G logs <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 853076deee223f9bd3c65a85b36fa766b4993666)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=853076deee223f9bd3c65a85b36fa766b4993666" class="external-link" target="_blank" rel="nofollow noopener">853076deee223f9bd3c65a85b36fa766b4993666</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/mgs/mgs_nids.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 853076deee223f9bd3c65a85b36fa766b4993666)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=853076deee223f9bd3c65a85b36fa766b4993666" class="external-link" target="_blank" rel="nofollow noopener">853076deee223f9bd3c65a85b36fa766b4993666</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/mgs/mgs_nids.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 853076deee223f9bd3c65a85b36fa766b4993666)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=853076deee223f9bd3c65a85b36fa766b4993666" class="external-link" target="_blank" rel="nofollow noopener">853076deee223f9bd3c65a85b36fa766b4993666</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_nids.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 853076deee223f9bd3c65a85b36fa766b4993666)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=853076deee223f9bd3c65a85b36fa766b4993666" class="external-link" target="_blank" rel="nofollow noopener">853076deee223f9bd3c65a85b36fa766b4993666</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_nids.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 853076deee223f9bd3c65a85b36fa766b4993666)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=853076deee223f9bd3c65a85b36fa766b4993666" class="external-link" target="_blank" rel="nofollow noopener">853076deee223f9bd3c65a85b36fa766b4993666</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/mgs/mgs_nids.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 853076deee223f9bd3c65a85b36fa766b4993666)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=853076deee223f9bd3c65a85b36fa766b4993666" class="external-link" target="_blank" rel="nofollow noopener">853076deee223f9bd3c65a85b36fa766b4993666</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/mgs/mgs_nids.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> recovery: don't always swap nidtbl entries (Revision 853076deee223f9bd3c65a85b36fa766b4993666)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&a=commit&h=853076deee223f9bd3c65a85b36fa766b4993666" class="external-link" target="_blank" rel="nofollow noopener">853076deee223f9bd3c65a85b36fa766b4993666</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_nids.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>The b2_2 version of change #2410 is at <a href="http://review.whamcloud.com/#change,3008" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3008</a>.</p> LU-1701 Development Rank 1|hzv6l3: Rank (Obsolete) 4601 Severity