https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5692 Lustre <p>We have a Lustre 2.4.2 file-system. It was upgraded from 1.8.7 without reformatting MDT/OST.</p> <p>Recently, we decided to upgrade it to 2.5.3.</p> <p>Prior to this upgrade, the file-system had clients running 2.4.2 and 2.5.3 at different time and running without problem. 4 clients only.</p> <p>Yesterday, we upgraded the server to 2.5.3. MDS first. Client running 2.4.2 hung for 15+ ,minutes. We then rebooted client and mount still hung. </p> <p>We then try to mount with 2 2.5.3 clients and both of them crashed.</p> <p>At this point, only MDS was upgraded to 2.5.3 and rest of OSSs were still running 2.4.2.</p> <p>I am attaching MDS logs here. </p> <p>Please advice</p> Linux lustre-mds-8-0.local 2.6.32-431.23.3.el6_lustre.x86_64 #1 SMP Thu Aug 28 20:20:13 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux<br/> LU-5692

Lustre 2.5.3 client mounting Lustre 2.5.3 failed

Bug Major Resolved Done Jian Yu Haisong Cai sdsc Tue, 30 Sep 2014 23:52:54 +0000 Fri, 25 Mar 2016 20:34:25 +0000 Fri, 25 Mar 2016 20:34:25 +0000 Lustre 2.5.3 0 7 <p>Yu, Jian</p> <p>Could you please assist with this issue?</p> <p>Thanks</p> <p>Peter</p> <p>We do not test interoperability running MDS and OSS with different versions. Is there a particular reason you didn't upgrade the OSS at the same time? While that may not relate directly to your client problem, it introduces potential problems that could be easily avoided. </p> <p>Hi Andreas,</p> <p>The reason was for rolling upgrade. So eventually, we will upgrade every server to a same version of Lustre.<br/> But with some of the file-systems, there are as many as 32 OSS and 128 OSTs, upgrading them all will take sometime.</p> <p>thanks,<br/> Haisong</p> <p>Hi Haisong,</p> <p>Could you please gather the vmcore crash dump file for the Lustre 2.5.3 client and upload it to "uploads/<a href="https://jira.whamcloud.com/browse/LU-5692" title="Lustre 2.5.3 client mounting Lustre 2.5.3 failed" class="issue-link" data-issue-key="LU-5692"><del>LU-5692</del></a>" directory on ftp.hpdd.intel.com? Thanks!</p> <p>I did an experiment on a small test cluster (2 Clients, 1 MGS/MDS, 1 OSS) with the following steps:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1. setup and start Lustre 1.8.8-wc1 filesystem 2. shutdown the entire Lustre 1.8.8-wc1 filesystem 3. clean upgrade all Lustre servers and clients at once to Lustre 2.4.2 4. start the entire Lustre 2.4.2 filesystem 5. run IOR and tar applications on the two live Lustre 2.4.2 Clients 6. rolling upgrade MGS/MDS to Lustre 2.5.3 7. rolling upgrade one Client to Lustre 2.5.3 8. rolling upgrade the other Client to Lustre 2.5.3 9. run IOR and tar applications on the two live Lustre 2.5.3 Clients 10. rolling upgrade OSS to Lustre 2.5.3 11. run IOR and Simul tests on the upgraded Lustre 2.5.3 filesystem </pre> </div></div> <p>I tried to provision Lustre 1.8.7-wc1 servers but got kernel panic failure caused by isci module, so I switched to use Lustre 1.8.8-wc1. All of the above steps passed testing.</p> <p>Thanks you for the information.</p> <p>Our file-system was upgraded from 1.8.7. I guess we are out of luck for rolling upgrade. </p> <p>By the way, we have reconfigured failed file-system for other purpose therefore unable to get vmcore dump produced for the time being.</p> <p>Haisong</p> <p>Hello Haisong,</p> <p>We are marking this one as resolved/done.</p> <p>If you need any further work done on this ticket, please let us know and we can re-open it.</p> <p>Thanks,<br/> ~ jfc.</p> Development Rank 1|hzwxgv: Rank (Obsolete) 15934 Severity