https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12098 Lustre <p>After lctl changelog_deregister, changelogs were not fully cleared:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@porteri:~]# pdsh -g mds 'lctl get_param "*.*.changelog_users"' | dshbak -c ---------------- eporter81 ---------------- mdd.lustre3-MDT0000.changelog_users=current index: 3856956771 ID index ---------------- eporter82 ---------------- mdd.lustre3-MDT0001.changelog_users=current index: 5663390742 ID index [root@porteri:~]# pdsh -g mds 'lctl get_param "*.*.changelog_size"' | dshbak -c ---------------- eporter81 ---------------- mdd.lustre3-MDT0000.changelog_size=15384080 ---------------- eporter82 ---------------- mdd.lustre3-MDT0001.changelog_size=25036880872 </pre> </div></div> Lustre 2.10.6_2.chaos <br/> zfs 0.7.11-5llnl <br/> distro toss-release-3.4-4 (RHEL 7.6 based) <br/> kernel 3.10.0-957.5.1.3chaos.ch6.x86_64 <br/> 2 MDTs, we use DNE1 <br/> See github.com/llnl/ for zfs and lustre code corresponding to these tags LU-12098

changelog_deregister appears not to reliably clear all changelog entries

Bug Minor Resolved Fixed Sebastien Buisson Olaf Faaland ORNL llnl Fri, 22 Mar 2019 00:27:36 +0000 Mon, 15 Jul 2019 21:53:36 +0000 Tue, 21 May 2019 14:39:04 +0000 Lustre 2.10.6 Lustre 2.13.0 Lustre 2.12.3 0 6 <p>For our tracking, our internal ticket is TOSS-4471</p> <p>I have not yet been able to test to see if it can be produced in master or 2.12. I will attempt that next week.</p> <p>In the case of one of MDT0001, the 'lctl deregister_changelog' command was issued and the MDS crashed. However, MDT0000 never went down.</p> <p>We can work around this in the future, on Lustre 2.10, by issuing an "lfs changelog_clear" first. For these MDTs, should I re-register a changelog user and then issue changelog_clear and changelog_deregister?</p> <p>Sébastien</p> <p>Could you please advise here?</p> <p>Thanks</p> <p>Peter</p> <p>Hi,</p> <p>My understanding is that changelog_deregister is supposed to issue a clear first, so there should be no difference if clear is not called before. And I am not sure registering a new changelog user to issue a clear and then deregister would help, because the newly created changelog user will have its index pointing to the latest existing index on the MDT, hence preventing to clear older indexes.</p> <p>The only way I can see to clear those remaining changelog records is to stop then restart the MDTs. Indeed, upon stop, the changelog record files are completely cleaned up.</p> <p>In your case, there might be a bug which prevents former llog files from being deleted while MDT is online. Changelogs records are split into multiple llog files on MDTs, each one containing up to around 16k records. Once an llog file becomes full (ie contains ~16k records), an additional llog file is created to store the next Changelog records. When a clear is issued for index N, all llog files for which maximum record index is lower than N are removed. Only remain the llog files that have records with index greater than or equal to N. If the llog file containing record N is the current, latest one, then it cannot be removed, because it is being used. No matter how many records are in there, the space consumed on the MDT by this llog file will be freed later, when a new clear is issued.</p> <p>In order to investigate, could you please access the impacted MDTs as ZFS, and run the following command to help finding the actual location of the changelog entries on the MDTs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># llog_reader /zfs_mount/changelog_catalog </pre> </div></div> <p>It should output something like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Header size : 8192 Time : Mon Mar 25 15:36:53 2019 Number of records: 4 Target uuid : ----------------------- #01 (064)id=[0x6:0x1:0x0]:0 path=oi.1/0x1:0x6:0x0 #02 (064)id=[0x7:0x1:0x0]:0 path=oi.1/0x1:0x7:0x0 #03 (064)id=[0x8:0x1:0x0]:0 path=oi.1/0x1:0x8:0x0 #04 (064)id=[0x9:0x1:0x0]:0 path=oi.1/0x1:0x9:0x0 </pre> </div></div> <p>What is important is the number of records (4 in this example), and the path to every llog file holding changelog records. In this example the files are oi.1/0x1:0x{6,7,8,9}:0x0.</p> <p>Then, for each of these files, it would be interesting to see how many records they contain:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># for i in 6 7 8 9; do llog_reader /zfs_mount/oi.1/0x1:0x${i}:0x0 | grep records ; done Number of records: 16577 Number of records: 16373 Number of records: 16320 Number of records: 10731 </pre> </div></div> <p>Thanks.</p> <blockquote><p>My understanding is that changelog_deregister is supposed to issue a clear first, so there should be no difference if clear is not called before.</p></blockquote> <p>That is my understanding too, but I believe the changelog_deregister first disables the changelog user. So if the MDT goes down while the changelog_deregister is occurring, how could one resume the clearing of the changelog records? I believe that they cannot be cleared at that point, at least by the user. But I am not certain of that.</p> <p>On the other hand, I believe the changelog_clear can just be invoked again after the MDT comes back up.</p> <p>I haven't had a machine to test either theory on, nor time to just walk the code to see, yet.</p> <blockquote><p>Indeed, upon stop, the changelog record files are completely cleaned up.</p></blockquote> <p>Do you recall where in the code this occurs? Is this for empty changelog record files (Number of records: 0) or for orphaned records like the ones we have?</p> <blockquote><p>Then, for each of these files, it would be interesting to see how many records they contain:</p></blockquote> <p>I verified all the changelog_catalog entries point to existing files, and all those files contain changelog records. I'll post more details later today.</p> <p>Here is a summary for each of the two MDTs. I counted the number of records of type "changelog" found within the files the changelog_catalog points to. MIN is the minimum number of records seen after looking at all the files; MAX is same except maximum number, and AVG and TOT are average and total (sum).</p> <p>Note that porter81 is hosting MDT 0 and porter82 is hosting MDT 1</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>---------------- eporter81 ---------------- Changelog_catalog header: Header size : 8192 Number of records: 2 Missing changelog files: 0 Existing changelog files: 2 checking oi.1/0x1:0x12b4a:0x0 checking oi.1/0x1:0x12b4b:0x0 all files checked MAX 59712 MIN 17606 TOT 77318 AVG 38659 ---------------- eporter82 ---------------- Changelog_catalog header: Header size : 8192 Number of records: 2911 Missing changelog files: 0 Existing changelog files: 2911 checking oi.1/0x1:0x2e14f:0x0 checking oi.1/0x1:0x2e150:0x0 &lt;... 2909 rows removed ...&gt; all files checked MAX 64101 MIN 11590 TOT 185533101 AVG 63735 </pre> </div></div> <p>Hi Sebastien,<br/> I tried stopping MDT 1 and then starting it again. changelog_size has not changed.</p> <p>Hi,</p> <p>changelog_deregister calls mdd_changelog_user_purge(). This one will invoke mdd_changelog_user_purge_cb(), which is responsible for (a) cancelling changelog records with indexes lower than that of the changelog user if not used by other user and (b) unregistering this changelog user. After that, mdd_changelog_user_purge() calls mdd_changelog_llog_cancel() to (c) purge changelog files if necessary.</p> <p>So you are right, if the MDS crashes in-between (b) and (c), you can end up with cancelled changelogs that are not freed. </p> <p>By introducing a new fail_loc for test purpose, I was able to make my MDS crash between step (b) and step (c). After reboot, the changelog user was not registered any more, but all its changelog entries were still on the MDT. So I registered a new changelog user, then immediately deregistered it, and it successfully cleared the pending changelog entries.</p> <p>It confirm that this technique is valid to get rid of remaining changelog entries.<br/> It does not work to call changelog_clear after the MDS comes back up, because the changelog user is not known anymore by the MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs changelog_clear lustre-MDT0001 cl1 0 lfs changelog_clear: cannot purge records for 'cl1': No such file or directory (2) changelog_clear: no changelog user: cl1 </pre> </div></div> <p>&gt;&gt; Indeed, upon stop, the changelog record files are completely cleaned up.<br/> &gt; Do you recall where in the code this occurs? Is this for empty changelog record files (Number of records: 0) or for orphaned records like the ones we have?</p> <p>The code path is the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdd_device_shutdown mdd_changelog_fini llog_cat_close llog_destroy </pre> </div></div> <p>So when an MDT is unmounted, llog_destroy is called for Changelog files that are referenced in a catalog file, and for which the number of records they contain is set to 0.<br/> It is not a cleanup of orphan objects.</p> <p>The information you gave with 'lctl get_param "<b>.</b>.changelog_users"' and 'lctl get_param "<b>.</b>.changelog_size"' on the one hand, and the information contained in the catalog and changelogs files (number of records, etc) on the other hand, is consistent with what I have when using my reproducer that crashes between steps (b) and (c).</p> <p>To sum up the situation, no more changelog users are registered, but the catalog file points to valid changelog files that are full of entries ('Number of records:' is not 0).</p> <p>On my test system I was able to get rid of these remaining changelogs not used anymore by any user by registering a new changelogs user and then deregistering it.<br/> Before registering:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param mdd.seb-MDT000*.changelog_size mdd.seb-MDT0000.changelog_size=0 mdd.seb-MDT0001.changelog_size=13186960 </pre> </div></div> <p>After de-registering:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param mdd.seb-MDT000*.changelog_size mdd.seb-MDT0000.changelog_size=0 mdd.seb-MDT0001.changelog_size=25384 </pre> </div></div> <p>Sebastien,<br/> Thank you. I'll do that next week, probably on Wednesday.</p> <p>Hi Sebastien,<br/> On both the affected file systems, the register/de-register procedure worked. Thank you.</p> <p>Are you planning to re-order the de-register code to prevent this problem?</p> <p>Hi Olaf,</p> <p>This is very good news!</p> <p>Regarding the way to solve this issue, the current implementation might be imposed by concurrency and/or deadlock concerns. So I was thinking about modifying 'changelog_deregister' so that it calls internally the 'changelog_clear' routines <b>before</b> actually proceeding to the consumer deregister. This way, if a crash occurs during the process, we could not end up with changelog entries not cleared but changelog consumer deregistered.</p> <p>I will try to push a patch for this.</p> <p>Thanks,<br/> Sebastien.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/34688" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34688</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12098" title="changelog_deregister appears not to reliably clear all changelog entries" class="issue-link" data-issue-key="LU-12098"><del>LU-12098</del></a> mdd: explicitly clear changelogs on deregister<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 21bd3ab61062d5c06a3a231ff7dbadb44dfb337a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34688/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34688/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12098" title="changelog_deregister appears not to reliably clear all changelog entries" class="issue-link" data-issue-key="LU-12098"><del>LU-12098</del></a> mdd: explicitly clear changelogs on deregister<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 83ffa859bc629e246de9fcdfc82838b14c6d0ea3</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34921" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34921</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12098" title="changelog_deregister appears not to reliably clear all changelog entries" class="issue-link" data-issue-key="LU-12098"><del>LU-12098</del></a> mdd: explicitly clear changelogs on deregister<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7904fb7e243042bce0fad2f2ac9f9fef33236e58</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34921/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34921/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12098" title="changelog_deregister appears not to reliably clear all changelog entries" class="issue-link" data-issue-key="LU-12098"><del>LU-12098</del></a> mdd: explicitly clear changelogs on deregister<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 5bf8c0eaf9101f98b49029a8651b73bce436db17</p> Related LU-11205 Development Rank 1|i00dpb: Rank (Obsolete) 9223372036854775807 Severity