https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14958 Lustre <p>revoke hash table can be enourmous with multi-GB journal. this may result in millions of revoke records which are loaded and inserted into specific hashtable during journal replay.<br/> currently the revoke hashtable's size is hard-coded as 256, thus every slot may get too many records.</p> <p>a simple benchmark of that code:<br/> 1048576 - 95 seconds<br/> 2097152 - 580 seconds<br/> in the fields it can be upto 30M records to find/insert.</p> <p>with 8192 buckets in the hash table:<br/> 4194304 - 59 seconds<br/> 8388608 - 247 seconds</p> LU-14958

configurable hash table size for jbd2

Improvement Minor Resolved Fixed Alex Zhuravlev Alex Zhuravlev Mon, 23 Aug 2021 19:40:19 +0000 Fri, 15 Sep 2023 04:13:23 +0000 Sat, 29 Apr 2023 03:14:34 +0000 Lustre 2.16.0 Lustre 2.15.3 0 9 <p>same benchmark with rhashtable:</p> <p>1048576 in 1854304 (2 s)<br/> 2097152 in 3750792 (3 s)<br/> 4194304 in 7594669 (7 s)<br/> 8388608 in 15385228 (15 s)<br/> 16777216 in 31040219 (31 s)<br/> 33554432 in 62284580 (62 s)</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pjones" class="user-hover" rel="pjones">pjones</a>, I'm waiting for a feedback from linux-ext4@ and then I plan to make patches for master</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45122" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45122</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14958" title="configurable hash table size for jbd2" class="issue-link" data-issue-key="LU-14958"><del>LU-14958</del></a> kernel: use rhashtable for revoke records in jbd2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cfdf255409758332bb0fc406c8b86578bbf6070b</p> <p>I did some measurements and tend to think the current approach should be changed a bit. I ran sanity-benchmark, it took 1724 seconds, all lookups in revoke table were counted for runtime (in contrast with journal replay):<br/> dm-0-8: 4666 lookups 7637 insertions in revoke table<br/> only 7637 lookups and 4666 insertions over 1700 seconds on MDS. currently insertion is simple as:<br/> spin_lock(&amp;journal-&gt;j_revoke_lock);<br/> list_add(&amp;record-&gt;hash, hash_list);<br/> spin_unlock(&amp;journal-&gt;j_revoke_lock);<br/> and this must be cheaper than an insertion into rhashtable which does lookup internally. I think there will be runtime performance penalty if we switch to rhashtable. probably we need a more flexible schema when rhashtable is used during journal replay and a simple list during regular operations.</p> <p>Hello,</p> <p>The CEA hit that issue on a ClusterStore version "2.12.4.2_cray" with a 4G journal:</p> <p>Robinhood stops to dequeue changelogs because of <a href="https://jira.whamcloud.com/browse/LU-14158" title="lfs changelog do not display old changelog after changelog_catalog wrapped arround" class="issue-link" data-issue-key="LU-14158"><del>LU-14158</del></a>. After one night changelog_catalog was 50% full.<br/> We try to deregister the changelog but the cancel process was too long (<a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a>). So we decide to crash the MDT.<br/> We were unable to mount the target. We identify that mount process got stuck in journal recovery with "perf" (find_revoke_record).<br/> We try e2fsck and it got stuck in journal recovery too. We identify (with strace) that the recovery process is not stuck but too slow (several days)</p> <p>To get back the filesystem in production quickly (with current IO losses):</p> <ul> <li>we save the raw journal into a file</li> <li>remove the need_recovery and has_journal flag</li> <li>e2fsck -f to apply the journal removing and correct inconsistencies</li> <li>reapply the has_journal ext4 flag to recreate the journal</li> <li>kill/crash all the current jobs (current IOs will be lost)</li> <li>mount lustre</li> <li>execute an LFSCK on the MDT to correct lustre inconsistencies</li> </ul> <p>There was &gt;64M revoke entries inside the jbd2 journal.<br/> For now we think this is the changelog_deregister that causes the creation of so many revoke records.</p> <p>I successfully reproduced the issue on a 2.12.7 LTS with changelog_deregister and with a 1228M journal:</p> <p>I executed changelog_deregister with 17794/64768 entries in changelog_catalog (27%, 17794 llog plain files).<br/> The revoke records appeared to increase linearly while removing llog plain: 2150 revoke_entries/s with a removing speed of -56 llog/min<br/> When I crashed the MDS there was ~20M of revoke entries in the journal, the journal recovery took about 40min. </p> <p>many thanks for the report, I'm going to try this locally.</p> <p>I retest the case with a 2.12.7 + <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a>'s patch:</p> <p>I executed changelog_deregister with 17794/64768 entries in changelog_catalog (27%, 17794 llog plain files).<br/> The revoke records grow linearly while removing llog plain: 109 revoke_entries/s with a removing speed of -2160 llog/min<br/> When I crashed the MDS there was ~50k of revoke entries in the journal, the journal recovery took about 1s. </p> <p>So the <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a> seems to resolve the issue for "changelog_deregister".</p> <p>in case you have time/wish, could you please try to collect more data? with NO <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a> patch make changelog_deregister with a very high number of changelog records (like in your example) and dump cat /proc/fs/jbd2/&lt;mds device&gt;/info just before crashing MDS?<br/> the revoke records are generated when metadata blocks are being freed, so there must be same number of revoke records but time distribution is very different, I guess.</p> <p>Hello,<br/> I have retested without the <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a> (it is same set of changelogs each time: I untar an archive to restore changelog state).<br/> This time, I waited the end of the deregister (like in the test with <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a>). It took 154min to complete.<br/> Here the "/proc/fs/jbd2/info" before the crash:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1858 transactions (1850 requested), each up to 262144 blocks average: 0ms waiting for transaction 0ms request delay 4988ms running transaction 8ms transaction was being locked 0ms flushing data (in ordered mode) 32ms logging transaction 43059us average transaction commit time 392638 handles per transaction 49 blocks per transaction 50 logged blocks per transaction </pre> </div></div> <p>Before the crash there was 30489150 revoke records in the on disk journal (count with "logdump -a", see <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/41380/41380_jbd2_debugfs.gz" title="jbd2_debugfs.gz attached to LU-14958">jbd2_debugfs.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>).<br/> The recovery took 98min.</p> <p>I added a "perf probe" on insert_revoke_hash() and I record the some events during the deregister:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 100.00% 100.00% (ffffffffc0d3ea10) | ---__GI___ioctl system_call_fastpath+37 sys_ioctl+161 do_vfs_ioctl+928 obd_class_ioctl+210 class_handle_ioctl+6349 mdt_iocontrol+1516 mdd_iocontrol+966 mdd_changelog_clear+1619 mdd_changelog_llog_cancel+208 llog_changelog_cancel+84 llog_cat_process+46 llog_cat_process_or_fork+481 llog_process_or_fork+188 llog_process_thread+2143 llog_cat_process_cb+151 llog_handle_put+72 llog_osd_close+78 lu_object_put+197 lu_object_free+104 osd_object_delete+468 iput+252 evict+180 ldiskfs_evict_inode+1418 ldiskfs_truncate+784 | |--60.87%--ldiskfs_ind_truncate+689 | | | --60.83%--ldiskfs_free_branches | | | --60.75%--ldiskfs_free_branches+247 | ldiskfs_free_branches+653 | | | |--36.95--ldiskfs_free_data+323 ldiskfs_clear_blocks+401 ldiskfs_free_blocks+411 __ldiskfs_forget+290 insert_revoke_hash+1 | | | | | | | | | | | --23.81%--ldiskfs_free_data+275 ldiskfs_clear_blocks+401 ldiskfs_free_blocks+411 __ldiskfs_forget+290 insert_revoke_hash+1 | --38.68%--ldiskfs_ind_truncate+785 | ldiskfs_free_branches+247 ldiskfs_free_branches+653 ldiskfs_free_data+323 ldiskfs_clear_blocks+401 ldiskfs_free_blocks+411 __ldiskfs_forget+290 insert_revoke_hash+1 </pre> </div></div> <p>Something I noticed is that the on disks revokes records number is stable after the deregister (even if we do some op on the MDT). It look like some kind of leak for me, but I don't understand when this records are supposed to be cancelled.</p> <p>Concerning the difference between the tests with and without the <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a>. At the end of the the changelog_deregister:</p> <ul> <li>with the <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a>: I have 50k on disk revoke records ( ~3 revoke records per llog plain)</li> <li>without the <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a>: I have 30M on disk revoke records ( ~1700 revoke records per llog plain)</li> </ul> <p>Here the "dumpe2fs -h" after recovery: <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/41381/41381_dumpe2fs.txt" title="dumpe2fs.txt attached to LU-14958">dumpe2fs.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <blockquote> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 1858 transactions (1850 requested), each up to 262144 blocks average: 0ms waiting <span class="code-keyword">for</span> transaction 0ms request delay 4988ms running transaction 8ms transaction was being locked 0ms flushing data (in ordered mode) 32ms logging transaction 43059us average transaction commit time 392638 handles per transaction 49 blocks per transaction 50 logged blocks per transaction </pre> </div></div></blockquote> <p>thanks a lot! and, if possible, the same with <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a> applied if you have few spare cycles. thanks in advance!</p> <p>I think this confirms the theory that basically "deregistiring" is CPU bound and produces a lot of tiny transaction which aren't checkpointed (I guess there is no need to do so - no memory pressure as number of modified blocks is tiny). given they aren't checkpointed by the crash, JBD has to replay them all and need to skip revoked blocks so fill/lookup big revoke table.<br/> the situation changes with <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a> as the process is less CPU-bound and generates bigger transactions and (somehow) transactions get checkpointed more frequently leaving less to replay.</p> <p>the important question here is whether we still need to fix JBD.. I tend to think so as there are another use cases when external processes (like llsom_sync) may consume and clear changelog at high rate and this would result in tiny transactions as before <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a> patch.</p> <p>Here the "/proc/fs/jbd2/info" with the <a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>131 transactions (127 requested), each up to 262144 blocks average: 0ms waiting for transaction 0ms request delay 4770ms running transaction 13ms transaction was being locked 0ms flushing data (in ordered mode) 60ms logging transaction 91593us average transaction commit time 826 handles per transaction 372 blocks per transaction 373 logged blocks per transaction </pre> </div></div> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/45122/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/45122/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14958" title="configurable hash table size for jbd2" class="issue-link" data-issue-key="LU-14958"><del>LU-14958</del></a> kernel: use rhashtable for revoke records in jbd2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c3bb2b778d6b40a5cecb01993b55fcc107305b4a</p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50730" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50730</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14958" title="configurable hash table size for jbd2" class="issue-link" data-issue-key="LU-14958"><del>LU-14958</del></a> kernel: use rhashtable for revoke records in jbd2<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 53e110a884423b137b44ea44b0f1327b1535bfa1</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50730/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50730/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14958" title="configurable hash table size for jbd2" class="issue-link" data-issue-key="LU-14958"><del>LU-14958</del></a> kernel: use rhashtable for revoke records in jbd2<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 52bc546e91e415315e6cf9a46608264122e64ef3</p> <p>Seems to be merged for 2.15.3 and 2.16</p> <p>It looks like this patch solved the problem with the kernel journal revoke record handling, but there is still a similar problem in e2fsprogs handling of journal replay. We hit an issue with slow journal recovery while deleting a large number of changelogs, and "<tt>tune2fs</tt>" and "<tt>e2fsck</tt>" were hung in journal recovery for hours before being interrupted. The problem was eventually fixed by updating server to include this fix in ldiskfs and then mounting the filesystem to do journal recovery in the kernel.</p> Related LU-17117 Development Rank 1|i022j3: Rank (Obsolete) 9223372036854775807