https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3929 Lustre <p>While performing rolling upgrade from Lustre 2.1.6 to 2.4.1 RC2 with the path of OSS-&gt;MDS-&gt;Client one by one, the test failed after upgrading MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Starting the MDS service on fat-amd-3... ---------------- fat-amd-3 ---------------- debug=-1 subsystem_debug=all -lnet -lnd -pinger debug_mb=100 pdsh -l root -t 100 -S -w fat-amd-3 "mkdir -p /mnt/mds1 &amp;&amp; mount -t lustre -o user_xattr /dev/sdc1 /mnt/mds1" Waiting 895 secs for fat-amd-3 recovery done. status: RECOVERING &lt;~snip~&gt; Waiting 5 secs for fat-amd-3 recovery done. status: RECOVERING Waiting 0 secs for fat-amd-3 recovery done. status: RECOVERING fat-amd-3 recovery not done in 900 sec. status: RECOVERING </pre> </div></div> <p>On MDS fat-amd-3, "lctl get_param -n <b>.</b>.recovery_status" showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>---------------- fat-amd-3 ---------------- status: RECOVERING recovery_start: 1378874775 time_remaining: 0 connected_clients: 2/4 req_replay_clients: 0 lock_repay_clients: 0 completed_clients: 2 evicted_clients: 0 replayed_requests: 0 queued_requests: 0 next_transno: 4294967297 </pre> </div></div> <p>Console log on MDS showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect Lustre: lustre-MDT0000: recovery is timed out, evict stale exports Lustre: lustre-MDT0000: disconnecting 2 stale clients Lustre: lustre-MDT0000: recovery is timed out, evict stale exports </pre> </div></div> <p>Maloo reports:<br/> <a href="https://maloo.whamcloud.com/test_sets/d91a2b68-1aa1-11e3-88ff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d91a2b68-1aa1-11e3-88ff-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/dae9450c-1a86-11e3-8ceb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dae9450c-1a86-11e3-8ceb-52540035b04c</a></p> <p>The same failure also occurred while rolling upgrade from Lustre 2.1.6 to 2.4.0:<br/> <a href="https://maloo.whamcloud.com/test_sets/c70af506-1ab5-11e3-8898-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c70af506-1ab5-11e3-8898-52540035b04c</a></p> LU-3929

2.1.6->2.4.1 rolling upgrade: lustre-MDT0000: recovery is timed out, evict stale exports

Bug Major Resolved Fixed Hongchao Zhang Jian Yu mn4 Wed, 11 Sep 2013 08:44:30 +0000 Mon, 7 Jul 2014 17:06:47 +0000 Mon, 6 Jan 2014 14:17:31 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.5.1 0 7 <p>Hongchao</p> <p>Could you please make an assessment of this issue?</p> <p>Thanks</p> <p>Peter</p> <p>this issue is related to the LWP(Light Weight Proxy) connection.<br/> in b2_1_*, the LWP connection is treated as normal clients and one client data "tg_export_data" is allocated for it, then there will be one more export<br/> to be recover during recovery at MDT, but in b2_4_*, MDT won't allocate client data for LWP connection, which doesn't need recover.</p> <p>Hi,</p> <p>We are suffering from this error, which is very annoying in case a customer wants to upgrade its OSSes first and then its MDSes and clients.</p> <p>This ticket was opened a month ago, but did not make any progress since then. This is surprising, as I would tend to consider it a major issue on the upgrade path from 2.1 to 2.4 (and 2.5 too). Am I missing something?</p> <p>Sebastien.</p> <p>status update:<br/> the patch is under testing and will be pushed to Gerrit soon. Thanks</p> <p>the patch is against b2_1, and is tracked at <a href="http://review.whamcloud.com/#/c/8086/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8086/</a></p> <p>the patch against master is tracked at <a href="http://review.whamcloud.com/#/c/8328/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8328/</a></p> <p>Hi, I have just tested patch <a href="http://review.whamcloud.com/#/c/8086/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8086/</a> for b2_1, and it works fine. I mean rolling upgrade from Lustre 2.1.6 plus this patch to 2.4.1 went off smoothly.</p> <p>So now I am wondering what is the purpose of this new patch <a href="http://review.whamcloud.com/#/c/8328/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8328/</a> for master.</p> <p>Sebastien.</p> <p>it could allow more previous Lustre version to upgrade to new version with the patch against master.</p> <p>Do you mean the master patch <b>alone</b> would be enough to be able to successfully upgrade from 2.1 with the path OSS-&gt;MDS-&gt;Client?</p> <p>Yes, the master patch alone should be enough to allow upgrades from unpatched 2.1 OSTS (i.e. those that do not have 8086 patch present). Can you give such a combination a try, please?</p> <p>We believe it's a better way since it saves you one extra step of upgrading all your OSTS to 2.1.6+patch before you can update your MDS to 2.4+ and then update your OSTs again to 2.4+ too (which is kind of overkill).</p> <p>Hi,</p> <p>Here is the test I carried out:</p> <ul class="alternate" type="square"> <li>full file system installed with stock 2.1.6</li> <li>upgrade to 2.4.1 + patch <a href="http://review.whamcloud.com/8328" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8328</a> with the path OSS-&gt;MDS-&gt;Client</li> </ul> <p>It went off smoothly. So I confirm that the master patch is enough. And, as Oleg explained, having the patch in the target version simplifies upgrade.</p> <p>Cheers,<br/> Sebastien.</p> <p>Landed for 2.5.1</p> Related LU-5298 Development Rank 1|hzw1u7: Rank (Obsolete) 10379 Severity