https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3446 Lustre <p>After a maintenance downtime in which we updated our vesta Lustre servers from 2.4.0-RC1_3chaos to 2.4.0-RC2_2chaos, the changelog current index reverted to zero. The registered user cl1 retained its highest index value:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># vesta-mds1 /root &gt; lctl get_param mdd.fsv MDT0000.changelog_users mdd.fsv-MDT0000.changelog_users= current index: 9158128 ID index cl1 445461827 </pre> </div></div> <p>Here is a debug log entry from when the MDS started:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000080:1.0:1370534969.929148:0:17901:0:(mdd_device.c:308:mdd_changelog_llog_init()) changelog starting index=0 </pre> </div></div> <p>LLNL-bug-id: TOSS-2103</p> mds: lustre-2.4.0-RC2_2chaos_2.6.32_358.6.1.3chaos.ch5.1.ch5.1.x86_64 LU-3446

changelog index reset on MDT restart

Bug Critical Resolved Fixed Zhenyu Xu Ned Bass HB changelog Fri, 7 Jun 2013 23:52:51 +0000 Tue, 20 Aug 2013 20:24:23 +0000 Tue, 20 Aug 2013 20:24:22 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 0 8 <p>Bobijam</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>the mdd_device.c:308 is from mdd initialize LLOG_CHANGELOG_ORIG_CTXT, after that, we come to LLOG_CHANGELOG_USER_ORIG_CTXT initialization, where in changelog_user_init_cb(), it sets mdd-&gt;mdd_cl.mc_lastuser, then we go back to mdd_change_llog_init()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* If we have registered users, assume we want changelogs on */</span> <span class="code-keyword">if</span> (mdd-&gt;mdd_cl.mc_lastuser &gt; 0) { rc = mdd_changelog_on(env, mdd, 1); <span class="code-keyword">if</span> (rc &lt; 0) GOTO(out_uclose, rc); } </pre> </div></div> <p>And mdd_changelog_on() <del>&gt; mdd_changelog_write_head(), it adds back mdd</del>&gt;mdd_cl.mc_index.</p> <p>Bobijam, does your comment indicate that this should not be an actual problem? Do we have a ChangeLog test that verifies the ChangeLog index after a restart?</p> <p>Ned,</p> <p>When did the cl1 changelog_users read happen, after the MDT restart?</p> <p>Here's my local VM situation.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10:38 AM]root@test1:~/work/lustre/lustre/tests/ # lctl get_param mdd.lustre-MDT0000.changelog_users mdd.lustre-MDT0000.changelog_users=current index: 58 ID index cl1 0 cl2 37 [10:42 AM]root@test1:~/work/lustre/lustre/tests/ # umount /mnt/mds1 [10:43 AM]root@test1:~/work/lustre/lustre/tests/ # mount -t lustre /dev/sdb /mnt/mds1 [10:43 AM]root@test1:~/work/lustre/lustre/tests/ # lctl get_param mdd.lustre-MDT0000.changelog_users mdd.lustre-MDT0000.changelog_users=current index: 58 ID index cl1 0 cl2 37 </pre> </div></div> <p>Bobijam, yes, the cl1 changelog_users users in the description happened after the MDT restart.</p> <p>Before this update, the current index was preserved between MDT restarts (as far as I know).</p> <p>We had a reboot yesterday, and, now that I check, it appears that the index was again reset:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># vesta-mds1 /root &gt; lctl get_param mdd.fsv-MDT0000.changelog_users mdd.fsv-MDT0000.changelog_users= current index: 444830 ID index cl1 445461827 cl2 18248038 </pre> </div></div> <p>It seems that if the registered changelog user has cleared all changelog records when the MDT is stopped, then when the MDT restarts <tt>changelog_init_cb()</tt> is never called from <tt>llog_reverse_process()</tt>, so <tt>mdd-&gt;mdd_cl.mc_index</tt> is left with 0.</p> <p>This reproduces the problem in a VM for me:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>FSTYPE=zfs llmount.sh lctl --device lustre-MDT0000 changelog_register touch /mnt/lustre/{1,2,3,4} lfs changelog_clear lustre-MDT0000 cl1 4 umount /mnt/mds1 mount.lustre lustre-mdt1/mdt1 /mnt/mds1 lctl get_param mdd.lustre-MDT0000.changelog_users </pre> </div></div> <p>Path for master:</p> <p><a href="http://review.whamcloud.com/#change,6642" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6642</a></p> <p>This needs to be fixed for 2.4.1 as well, so you should really added that Fix Version as well.</p> <p>When should we expect to see the patch for 2.4.1?</p> <p>Patch for b2_4:</p> <p><a href="http://review.whamcloud.com/#/c/7012/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7012/</a></p> <p>Landed for 2.4.1 and 2.5</p> Development Rank 1|hzvsxr: Rank (Obsolete) 8596 Severity