https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1543 Lustre <p>Dear Support, <br/> during the weeks we had problem with the MDS/OSS Lustre servers, indeed last week one of the OSS died and fortunately the fail over took over and last night both MDS (see log weissh01.log) and another OSS (see log weiss11.log) died, also in that case the fail over servers took over. <br/> The problem of yesterday looks related between the two servers, indeed they basically died at the same time around 19.20-30 </p> <p>As I said the file system remained up and running because the fail over servers took over, but with our old lustre configuration (version 1.8.7 &#8211; 1 Mds + 1 Mds Fail over &#8211; 4 Oss) also under huge stress and a lot of logging of slow down due to heavy IO load, the MDS or OSS didn't died.</p> <p>If you need access to our cluster, please let me know (fverzell@cscs.ch) so that we can organize to create an account.</p> <p>Right now we have also a list of ticket that might be related to each other in same aspect, that's the list:</p> <p><a href="http://jira.whamcloud.com/browse/LU-1447" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1447</a><br/> <a href="http://jira.whamcloud.com/browse/LU-1455" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1455</a><br/> <a href="http://jira.whamcloud.com/browse/LU-1470" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1470</a><br/> <a href="http://jira.whamcloud.com/browse/LU-1503" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1503</a></p> <p>Regards<br/> Fabio</p> MDS HW <br/> ---------------------------------------------------------------------------------------------------- <br/> Linux XXXX.admin.cscs.ch 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 <br/> Architecture: x86_64 <br/> CPU op-mode(s): 32-bit, 64-bit <br/> Byte Order: Little Endian <br/> CPU(s): 16 <br/> Vendor ID: AuthenticAMD <br/> CPU family: 16 <br/> 64Gb RAM <br/> Interconnect IB 40Gb/s <br/> <br/> MDT LSI 5480 Pikes Peak <br/> SSDs SLC <br/> ---------------------------------------------------------------------------------------------------- <br/> <br/> OSS HW <br/> ---------------------------------------------------------------------------------------------------- <br/> Architecture: x86_64 <br/> CPU op-mode(s): 32-bit, 64-bit <br/> Byte Order: Little Endian <br/> CPU(s): 32 <br/> Vendor ID: GenuineIntel <br/> CPU family: 6 <br/> 64Gb RAM <br/> Interconnect IB 40Gb/s <br/> <br/> OST LSI 7900 <br/> ---------------------------------------------------------------------------------------------------- <br/> <br/> Router nodes <br/> ------------------- <br/> 12 router nodes - IB 40Gb/s <br/> <br/> Clients <br/> --------- <br/> Cray XE6 - Lustre 1.8.6 <br/> <br/> <br/> <br/> 1 MDS + 1 fail over <br/> 12 OSS - 6 OST per OSS LU-1543

Lustre Servers - MDS / OSS Died & fail over took over

Bug Major Resolved Fixed Cliff White Fabio Verzelloni Wed, 20 Jun 2012 02:27:49 +0000 Mon, 10 Sep 2012 13:28:35 +0000 Mon, 10 Sep 2012 13:28:35 +0000 Lustre 2.2.0 0 3 <p>Fabio</p> <p>We will definitely take an overall view of all your issues when deciding the best approach. Getting remote access to the cluster in question will undoubtedly be useful. I will contact you directly to make those arrangements</p> <p>Peter</p> <p>Can we get a list of the address for the Lustre servers? </p> <p>I think it could be a dup of <a href="https://jira.whamcloud.com/browse/LU-1280" title="kernel BUG at .../lustre-2.1.0/lustre/lvfs/fsfilt-ldiskfs.c:978" class="issue-link" data-issue-key="LU-1280"><del>LU-1280</del></a> and fixes:<br/> <a href="http://review.whamcloud.com/#change,2452" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2452</a><br/> <a href="http://review.whamcloud.com/#change,2827" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2827</a></p> <p>The list of the Lustre servers is the following:</p> <p>MDS + Failover<br/> Weisshorn01- 148.187.7.101<br/> Weisshorn02- 148.187.7.102</p> <p>OSS + each couple is the failover (weisshorn03-04, 05-06, ecc..)<br/> Weisshorn03- 148.187.7.103<br/> Weisshorn04- 148.187.7.104<br/> Weisshorn05- 148.187.7.105<br/> Weisshorn06- 148.187.7.106<br/> Weisshorn07- 148.187.7.107<br/> Weisshorn08- 148.187.7.108<br/> Weisshorn09- 148.187.7.109<br/> Weisshorn10- 148.187.7.110<br/> Weisshorn11- 148.187.7.111<br/> Weisshorn12- 148.187.7.112<br/> Weisshorn13- 148.187.7.113<br/> Weisshorn14- 148.187.7.114</p> <p>Has there been any word from Cray on access to gnilnd source? Should we close this issue and revisit after the software version change planned (<a href="https://jira.whamcloud.com/browse/LU-1503" title="Clients application IO errors and overloaded system messages" class="issue-link" data-issue-key="LU-1503"><del>LU-1503</del></a>) ?<br/> Please let us know what more we can do to assist.</p> <p>Cliff, <br/> we passing the request to CRAY to see if can manage to have access to the code. I'll let you ASAP.</p> <p>Thanks<br/> Fabio</p> <p>FYI, Cray is working on pushing up the gnilnd into the Lustre tree. The tracking ticket is <a href="https://jira.whamcloud.com/browse/LU-1419" title="Tracking ticket for gnilnd push" class="issue-link" data-issue-key="LU-1419"><del>LU-1419</del></a>.</p> <p>Any updates?</p> <p>Well, the Cray LND code has been pushed to <a href="https://jira.whamcloud.com/browse/LU-1419" title="Tracking ticket for gnilnd push" class="issue-link" data-issue-key="LU-1419"><del>LU-1419</del></a>, but not landed. I can't help with any other updates.</p> <p>I mean does Fabio still see the problem.</p> <p>What is the current state? Is there anything more we can do on this issue?</p> <p>I am going to close this issue. Please re-open if you have more information or questions.</p> Development Rank 1|hzvguf: Rank (Obsolete) 6378 Severity