https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8465 Lustre <p>Mentioned in <a href="https://jira.whamcloud.com/browse/LU-7592" title="Сhange force_over_128tb lustre mount option to force_over_256b for ldiskfs" class="issue-link" data-issue-key="LU-7592"><del>LU-7592</del></a> by Andreas Dilger</p> <p>e2fsck performance will become an issue at this scale, and it would likely need to be parallelized to be able to complete in a reasonable time. It could reasonably expect multiple disks at this scale, so having larger numbers of IOs in flight would help, as would an event-driven model with aio that generates lists of blocks to check (itable blocks first), submits them to disk, and then processes them as they are read, generating more blocks to read (more itable blocks, indirect/index/xattr/directory blocks, etc), repeat.</p> LU-8465

parallel e2fsck performance at scale

Improvement Major Resolved Fixed Wang Shilong Artem Blagodarenko Tue, 2 Aug 2016 09:41:20 +0000 Fri, 26 Nov 2021 21:36:44 +0000 Wed, 30 Sep 2020 00:18:02 +0000 Lustre 2.14.0 0 14 <p>I think that before you start looking at format changes to improve e2fsck,<br/> that you look at what can be done to improve e2fsck performance on existing<br/> filesystems.  I think there is a lot that can be done, especially with SSD<br/> or RAID-10 storage that is mostly idle during the scan.  This has the benefit<br/> that it helps all filesystems, instead of only new ones, and improvements<br/> that do not change the disk format are more easily merged.</p> <p>There was work done in 1.42 or 1.43 to do async readahead of inode tables<br/> and other blocks, which improved e2fsck performance, but I suspect there<br/> is more that could be done in this area.</p> <p>Firstly, an example e2fsck run on my home OST and MDT filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>myth-OST0000: clean, 359496/947456 files, 781092161/970194944 blocks e2fsck 1.42.13.wc5 (15-Apr-2016) Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 5116k/932k (4903k/214k), time: 181.93/ 4.12/ 1.84 Pass 1: I/O read: 160MB, write: 0MB, rate: 0.88MB/s Pass 2: Checking directory structure Pass 2: Memory used: 11532k/1084k (4614k/6919k), time:  3.51/ 2.59/ 0.12 Pass 2: I/O read: 37MB, write: 0MB, rate: 10.53MB/s Pass 3: Checking directory connectivity Pass 3A: Memory used: 11532k/1084k (4617k/6916k), time:  0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 11532k/1084k (4612k/6921k), time:  0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 225.43MB/s Pass 4: Checking reference counts Pass 4: Memory used: 5196k/932k (3932k/1265k), time:  0.09/ 0.09/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Pass 5: Memory used: 7680k/932k (3917k/3764k), time: 227.05/ 1.72/ 3.03 Pass 5: I/O read: 187MB, write: 0MB, rate: 0.82MB/s </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>myth-MDT0000: clean, 1872215/3932160 files, 824210/3932160 blocks e2fsck 1.42.13.wc5 (15-Apr-2016) Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 13092k/17500k (12876k/217k), time: 19.98/ 5.52/ 1.72 Pass 1: I/O read: 1104MB, write: 0MB, rate: 55.26MB/s Pass 2: Checking directory structure Pass 2: Memory used: 21012k/8468k (13732k/7281k), time: 23.28/ 7.54/ 2.25 Pass 2: I/O read: 1113MB, write: 0MB, rate: 47.81MB/s Pass 3: Checking directory connectivity Pass 3A: Memory used: 21624k/8468k (14212k/7413k), time:  0.00/ 0.01/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 21624k/5572k (13731k/7893k), time:  0.06/ 0.05/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 15.44MB/s Pass 4: Checking reference counts Pass 4: Memory used: 21624k/484k (2638k/18987k), time:  0.42/ 0.45/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Pass 5: Memory used: 21624k/0k (2637k/18988k), time:  1.30/ 0.38/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 0.77MB/s </pre> </div></div> <p>It shows that a large amount of time is spent in pass1 (inode/block scan).<br/> The inode checking could be almost trivially multi-threaded, since each<br/> inode is independent.  It would need locking of shared structures.  I<br/> would make this an event-driven (producer/consumer) process, with threads<br/> doing prefetch reads of inode tables, then processing the first group<br/> on the shared list. This avoids threads sitting idle with a static<br/> division of block groups if some groups are used more than others.</p> <p>Pass2 (directory structure) is trivial on an OST because it has so few<br/> directories, but is relatively longer on the MDT (which is SSD based).<br/> During the pass 1 inode scan, a list of directory inodes will be generated,<br/> which could start prefetching some directory leaf blocks to do leaf block<br/> scanning. Again, this can be done largely in parallel as long as there is<br/> reasonable locking on the bitmaps and a few other structures.  It would<br/> be good to have some sort of elevator for async directory block readahead,<br/> to avoid random IO on HDD-based storage.</p> <p>Pass 5 is slow on the OST filesystem because it is HDD based and is not<br/> using flexbg, so there lots of seeking to read/update groups and bitmaps.<br/> That probably doesn't need to be multi-threaded on any modern filesystem.</p> <p>As for how to approach development, I'd start with a simple threading<br/> model (pthreads) and add locking to the bitmap and list structures (maybe<br/> a 64-slot hashed lock for bitmaps like include/linux/blockgroup_lock.h in<br/> the kernel, instead of one lock per bitmap block, since most are unused).<br/> For ease of acceptance, it should be possible to disable threading and<br/> locking at compile time for single-threaded use (e.g. embedded or non-linux).</p> <p>Then, a producer/consumer model for processing inode table blocks could<br/> just be split on a per-group basis with very little change to the code.<br/> Even just a simple counter for "last processed group" to track which<br/> group an idle thread should work on next might be enough.  These threads<br/> would also submit async inode table readahead of maybe 3*num_threads<br/> (about 1GB/1M inodes) to ensure threads are not waiting on disk IO.</p> <p>Finally, a separate producer/consumer model for processing directories<br/> that are found during pass1 could be added.  I'm not sure, without<br/> checking the code, whether pass 2 can be started before pass1 is done.</p> <p>With the addition of larger directory support, this would benefit from<br/> being able to shrink ext4 directories online, so that directories shrink<br/> as old entries are removed.  There have been many discussions on how to<br/> implement ext4 directory shrinking, so let me know if this is something<br/> you are interested in.</p> <p>Cheers, Andreas</p> <p>Artem, I know this is something you were interested in previously, have you done any work in this area?</p> <p>Andreas, not so far we faced with e2fsck slow work on one of our system. And finally make <a href="https://jira.whamcloud.com/browse/LU-11882" title="OST recreated objects gets badness mark from e2fsck" class="issue-link" data-issue-key="LU-11882"><del>LU-11882</del></a> patch. All other work is planned, but currently is suspended. </p> <p>Patch: <a href="https://review.whamcloud.com/#/c/35597/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35597/</a> cleanup e2fsck_pass1</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35671" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35671</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: cleanup abort in e2fsck_pass1_check_inode<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: f26dd61c2b2aad9cd7cda0df198fe3c8b7eed2a8</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35676" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35676</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: cleanup bad block in inode table in e2fsck_pass1_check_inode()<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 53959bf250cc036e77fc54b9417d04085839a1d1</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35678" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35678</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add ES_RESTART for enum e2fsck_status<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 37f60cc06b759172b5850dea5df55d9d47b2eb5d</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35684" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35684</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: cleanup struct e2fsck_struct<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: ea099b12438dd826b9719d8493287367370dd664</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35689" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35689</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add -m option for multithread<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: a458233a5dc11979dd08b8c74dc7785eb6dfc58a</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35690" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35690</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy context when using muti-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 16d2387e62a0f7b8c2e296f76f39590d57b0a537</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35691" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35691</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: debug<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: bd6215e5adbec633006bdf5490ca99505867e237</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35696" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35696</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy fs when using muti-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 3b384dc9e0bf7b98433d65c7138e013982de3d5b</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35697" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35697</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: debug<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 7eff9d589624330db9815ad710703b11425f0f20</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35698" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35698</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy dblist when using muti-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: da7c00d0421667363b1bc5a70498efd1553b0b5f</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35699" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35699</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: debug "copy dblist when using muti-thread fsck"<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: abb9dbc99651d37c2d43e4a0a0757b96007e9327</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35701" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35701</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: clear icache when using muti-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 02d35235d637e44ac0a6feee623adfd971e05cf3</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35702" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35702</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: debug "clear icache when using muti-thread fsck"<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 34f00c6c4429600cd4fe1734e2eaf6c61a675085</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35703" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35703</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: debug "clear icache when using muti-thread fsck" without cleanup<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: df7a0daccf9a6cb7e1939ef89585790d9f3da02b</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35710" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35710</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add assert when copying context<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 2d597965b393b5172208dd909b0c07658a7cd217</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35712" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35712</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy bitmaps when copying context<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: d5823b6137559d965162136bf281b8a0a834aae9</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35713" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35713</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: openfs rather than copying it<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 5bbaa98381535e853ac186daa18cd6da99194045</p> <p>The imeplementation of parallel fsck is going to very hard, not because of the new codes are hard, infact, new codes and logics are easy. The hardest part is how to add these codes without breaking any existing codes that exist and grow for decades. I am using several ways to confirm the patches don't break anything:</p> <p>TEST #1. The regression tests under "tests" directory. </p> <p>TEST #2. "valgrind" test to eliminate memory leak.</p> <p>TEST #3. e2fsck on a huge Ext4 with hundreds of millions inodes to confirm no performance regression.</p> <p>I am running all of these three tests to make sure every patch looks good. If you have any idea of other ways, please let me know and thank you so much!</p> <p>The design is to copy the fsck context for each pass1 thread. And each context is seperated with other ones. When the fsck thread finishes, the global context will merge all contexts together, and continue run the pass2...5 tests.</p> <p>And the plan is doing following steps one by one:</p> <p>#1. Cleanup the codes if possible. Unfortunately, it is hard. According to my experience, it is so easy to break something. For example, <a href="https://review.whamcloud.com/#/c/35597/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35597/</a> passed all regression tests, but the fsck is slown down dramatically.</p> <p>#2. Copy the context and its sub-fields to a new one, and run the same process like before. No thread will be created, and same functions are used even context is new one.</p> <p>#3. Copy the context to multiple ones, and split the pass1 scanning of inodes to multiple steps. Each scanning step works on a range of inodes. And this will still be done without multi-thread. Each step uses different context, and these contexts will be merged into the global one. The global context will be used to run the pass2...5 check.</p> <p>#4. Run all kinds of tests for step 3 by splitting the pass1 one scanning to different numbers of loops, from one to as large as the core number. This will make sure the context copying and merging process is correct.</p> <p>#5. Create only one thread to run the pass1 check in a new context. This should still be able to pass all above tests including TEST #1. And it is impossible to have race problem of disk read/write.</p> <p>#6. Create multiple threads to run the pass1 check in different threads. Each thread uses its own context, and works on a given range of inodes. This step is going to be very very hards because of races, including disk read/write conflict, global variable conflict and so on. So it will be necessary to split it into multiple steps:</p> <p>#6.1 Roughly implement the multi-thread fsck and run it on a clean file system to make sure the checking works well. Performance numbers can be got after this step is finished.</p> <p>#6.2 Implement the fully fsck and fix.</p> <p>After step #6, TEST #1 isn't going to pass because the output of fsck is going to be changed. But still, we can change the fsck tests to:</p> <p>FSCK TEST #1. Run fsck with multi-thread, but don't compare the result with the expected result of single thread.</p> <p>FSCK TEST #2. Run fsck without multi-thread, compare the result with the expected second result.</p> <p>All TEST #1 tests should be passed (maybe not for a few ones?). And this will make sure multi-thread fsck at least fixes the file system problems as expected.</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35726" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35726</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy badblocks when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 8e46169b9fa193f9ae9243a37eda824d4e0dcf08</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35731" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35731</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy mmap when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: cb0cb2a0402607afa6ae8bc343f6fbfe88793715</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35748" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35748</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy device name when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 46f170057b78417ebe009d91a41399ee47bf29e2</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35753" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35753</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy super when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 4b660e2fcc160b1c507c692118d31af0c6cf0286</p> <p>Is this being pushed upstream as well?</p> <p>Hi James, not yet. Will do when the performance numbers confirm the idea.</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35763" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35763</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: open io-channel when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: f2e83edde8df72a279951bbbb912fc7034a358ac</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35770" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35770</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy group_desc when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 2a5416da3a29337a294a6d2a988fc70881bbc1fb</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35771" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35771</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy image_header when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 6f49863cbe52d6b9f2a6453108b7e8b6ab1a2cc2</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35772" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35772</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: assert block_sha_map is NULL when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 3017502ad4f9bb2f176b4e029622f0dadb3fd64e</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35780" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35780</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: create logs for mult-threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: ecb338d8acc7d54efb3c99432540b910bf6ef7cd</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35793" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35793</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: create one thread to fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 8c65f9f1b89a157f38ccf0c876c6abef9e532966</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35846" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35846</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add start/end group for thread<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 4618d1ee83d0c60ce51a5cf2f435f58b75a436ae</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35855" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35855</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: split groups to different threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 820e9837cf6e5fc2a778be599c8deaf037f665a7</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35922" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35922</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add perf debug info<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: d5a8739048a3956d80df3f2b2ca8ce36fcb31b99</p> <p>On a 1PB Ext4 file system, I created 105M inodes. And run e2fsck:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># df | grep sda Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda 1099637648128 13563264 1088621733412 1% /mnt # df -i | grep sda Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda 1074397184 105887860 968509324 10% /mnt </pre> </div></div> <p>With original e2fsck:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e2fsck 1.45.2.wc1 (27-May-2019) MMP interval is 5 seconds and total wait time is 22 seconds. Please wait... Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 75116k/193852k (70544k/4573k), time: 3771.03/1472.81/31.63 Pass 1: I/O read: 52117MB, write: 0MB, rate: 13.82MB/s Time used: 3771 seconds Time used: 0 seconds Pass 2: Checking directory structure Pass 2: Memory used: 75116k/337408k (27172k/47945k), time: 98.10/86.34/ 3.37 Pass 2: I/O read: 2897MB, write: 0MB, rate: 29.53MB/s Time used: 98 seconds Pass 3: Checking directory connectivity Peak memory: Memory used: 75116k/337408k (27172k/47945k), time: 3879.43/1568.92/35.05 Pass 3: Memory used: 75116k/337408k (25930k/49187k), time: 0.02/ 0.03/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Time used: 0 seconds Pass 4: Checking reference counts Pass 4: Memory used: 75116k/0k (21993k/53124k), time: 12.94/12.95/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Time used: 13 seconds Pass 5: Checking group summary information Pass 5: Memory used: 75116k/0k (20108k/55009k), time: 66.93/ 4.12/ 0.13 Pass 5: I/O read: 206MB, write: 0MB, rate: 3.08MB/s Time used: 67 seconds 105887860 inodes used (9.86%, out of 1074397184) 1 non-contiguous file (0.0%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 105887853/1 139657888 blocks used (0.05%, out of 275045679104) 0 bad blocks 1 large file 105781892 regular files 105959 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 105887851 files Memory used: 75116k/0k (20108k/55009k), time: 3959.37/1586.01/35.18 I/O read: 55235MB, write: 0MB, rate: 13.95MB/s </pre> </div></div> <p>Using the e2fsck with patches <a href="https://review.whamcloud.com/#/c/35922/4" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35922/4</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e2fsck 1.45.2.wc1 (27-May-2019) MMP interval is 5 seconds and total wait time is 22 seconds. Please wait... iii thread 0 is going to scan group [0, 32788) [Thread 0] Pass 1: Checking inodes, blocks, and sizes iii thread 1 is going to scan group [32788, 65576) [Thread 1] Pass 1: Checking inodes, blocks, and sizes iii thread 2 is going to scan group [65576, 98364) [Thread 2] Pass 1: Checking inodes, blocks, and sizes iii thread 3 is going to scan group [98364, 131152) [Thread 3] Pass 1: Checking inodes, blocks, and sizes iii thread 4 is going to scan group [131152, 163940) [Thread 4] Pass 1: Checking inodes, blocks, and sizes iii thread 5 is going to scan group [163940, 196728) [Thread 5] Pass 1: Checking inodes, blocks, and sizes iii thread 6 is going to scan group [196728, 229516) [Thread 6] Pass 1: Checking inodes, blocks, and sizes iii thread 7 is going to scan group [229516, 262304) [Thread 7] Pass 1: Checking inodes, blocks, and sizes [Thread 4] Pass 1: Memory used: 205208k/1349632k (204375k/834k), time: 1407.46/11259.11/ 0.78 [Thread 4] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s [Thread 6] Pass 1: Memory used: 205276k/1349632k (203804k/1473k), time: 1407.57/11260.05/ 0.63 [Thread 6] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s [Thread 7] Pass 1: Memory used: 205308k/1349632k (203232k/2077k), time: 1407.66/11260.62/ 0.55 [Thread 7] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s [Thread 3] Pass 1: Memory used: 205376k/1349632k (202660k/2717k), time: 1407.81/11261.19/ 0.82 [Thread 3] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s [Thread 1] Pass 1: Memory used: 206752k/1349632k (202812k/3941k), time: 1408.34/11263.09/ 0.89 [Thread 1] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s [Thread 2] Pass 1: Memory used: 206820k/1349632k (202244k/4577k), time: 1408.45/11263.34/ 0.86 [Thread 2] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s [Thread 5] Pass 1: Memory used: 206892k/1349632k (201674k/5219k), time: 1408.51/11263.37/ 0.72 [Thread 5] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s [Thread 0] Pass 1: Memory used: 254040k/1374780k (245271k/8770k), time: 3805.86/11436.56/10.64 [Thread 0] Pass 1: I/O read: 52117MB, write: 0MB, rate: 13.69MB/s iii 0 scaned group [0, 32788) 105887861 inodes iii 1 scaned group [32788, 65576) 0 inodes iii 2 scaned group [65576, 98364) 0 inodes iii 3 scaned group [98364, 131152) 0 inodes iii 4 scaned group [131152, 163940) 0 inodes iii 5 scaned group [163940, 196728) 0 inodes iii 6 scaned group [196728, 229516) 0 inodes iii 7 scaned group [229516, 262304) 0 inodes Pass 2: Checking directory structure Pass 2: Memory used: 254040k/1543484k (108778k/145263k), time: 0.10/ 0.03/ 0.08 Pass 2: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 254040k/1543484k (108778k/145263k), time: 3816.89/11446.48/10.78 ... </pre> </div></div> <p>In most filesystems the inode distribution is uneven. Groups at the start of the filesystem have more inodes allocated than at the end. In the absence of dynamic group allocation, it probably makes more sense to do interleaved group allocation, like one flexbg of inodes (128-256 groups) to each thread in round-robin order. </p> <p>That can still be done without coordination between threads, but is more likely to load the threads evenly. It also is more likely to reduce seeking between threads. </p> <p>It would be useful for your testing to capture blocktrace data and plot with seekwatcher to see the IO patterns of the parallel e2fsck, as it may be the parallel COU improvement is being offset by worse IO patterns (pass1 was previously perfectly linear IO).</p> <p>Thank you Andreas for the advices. We found the root cause of the long time costs for the threads that had actually no work to do because of empty block groups. The patch <a href="https://review.whamcloud.com/#/c/35659/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35659/</a> can fix the problem. I am going to check whether there is anyway to get a good number with that patch. And then optimize the group blance between threads as you suggested.</p> <p>It isn't clear to me how patch 35659 ("libext2fs: optimize ext2fs_convert_subcluster_bitmap()") could fix this problem? AFAIK, that patch mostly relates to mke2fs.</p> <p>&gt; It isn't clear to me how patch 35659 ("libext2fs: optimize ext2fs_convert_subcluster_bitmap()") could fix this problem? AFAIK, that patch mostly relates to mke2fs.</p> <p>The pass1 check also calls ext2fs_convert_subcluster_bitmap(). The device has 1PB size, thus a lot of groups. But most of the groups are not used. ext2fs_convert_subcluster_bitmap() costs a lot of time. According to th e test result, each thread spends ~1400 seconds on this.</p> <p>After using the patch, the time cost have been reduced to about 10 seconds.</p> <p>Following is a test log with patch <a href="https://review.whamcloud.com/#/c/35922/8" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35922/8</a></p> <p>I used 128 threads to do the pass1 check. And only 13 threads have real work to do. It took at most 800 seconds for the slowest thread to finish.</p> <p>The time cost of pass1 check would take 3771 seconds. That means, it might be possible to get speed up of 3771/800 = 4.7 times.</p> <p>log: <a href="https://jira.whamcloud.com/secure/attachment/33433/128_threads.txt" class="external-link" rel="nofollow">https://jira.whamcloud.com/secure/attachment/33433/128_threads.txt</a></p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35961" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35961</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge bitmaps after thread completes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 06c6eca0e1447669e20a4cd9c10421610a4b1a9d</p> <blockquote> <p>I used 128 threads to do the pass1 check. And only 13 threads have real work to do. It took at most 800 seconds for the slowest thread to finish.</p></blockquote> <p>While there is a limit to the parallelism on a given system, if you split the workload into <tt>flexbg_count</tt> chunks (Lustre OSTs use flexbg_count=256 so that inode tables and block bitmaps are allocated in 1MB chunks on disk) then the workload could be spread over 100 threads in your test case, and the workload would likely be much more even. Each thread would process groups numbered:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>group = n * num_threads + flexbg_count * thread + [0..flexbg_count - 1], n: 0, 1, 2, ... </pre> </div></div> <p>In theory, even with the 13 threads doing an even amount of work (I'm assuming a "base" of ~215s for each thread, and the 5000s above that is "work") then pass1 would finish in about 635s. If 104 threads could participate evenly (at 256 groups/thread no more will have work to do), then pass1 could finish in about 265s.</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36001" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36001</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: do not change global variables<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: a8ef87bfc39ee35441f2763de44c154ddb818716</p> <p>Li Xi, have you looked at using pthreads for the process control and communication between threads rather than the current fork+exec?</p> <p>Looking at the e2fsck performance results, it seems that pass3 takes as long to complete as pass1, so it seems that once we have pass1 running in parallel then we also need to improve pass3 in order to significantly reduce the total runtime. </p> <p>With pass3 (pathname connectivity), it will definitely need more fine-grained communication between the threads to exchange results, and also also shared memory to avoid significant duplication of effort or memory usage. </p> <blockquote><p>Li Xi, have you looked at using pthreads for the process control and communication between threads rather than the current fork+exec?</p></blockquote> <p>Currently, my patches are already using pthreads, not fork+exec.</p> <blockquote><p>Looking at the e2fsck performance results, it seems that pass3 takes as long to complete as pass1, so it seems that once we have pass1 running in parallel then we also need to improve pass3 in order to significantly reduce the total runtime. </p></blockquote> <p>I don't think so. Pass3 only takes a 0.02 seconds according to <a href="https://jira.whamcloud.com/browse/LU-8465?focusedCommentId=253670&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-253670" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-8465?focusedCommentId=253670&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-253670</a></p> <p>The logs of e2fsck is not super clear though:</p> <p>Pass 3: Checking directory connectivity<br/> Peak memory: Memory used: 75116k/337408k (27172k/47945k), time: 3879.43/1568.92/35.05<br/> ^^^^^^^^^^^^^^^^ This is not the time/memory usage for Pass3, this is the time since fsck starts.<br/> Pass 3: Memory used: 75116k/337408k (25930k/49187k), time: 0.02/ 0.03/ 0.00<br/> ^^^^^^^^^^^^^^^^ This is is the time/memory usage for Pass3<br/> Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36004" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36004</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: optimize the inserting of dir_info_db<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 9022f19466a871305b77b7605dc968545b20f6a7</p> <p>Patch <a href="https://review.whamcloud.com/36004" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36004</a> adds e2fsck_dir_info_min_larger_equal(). <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/33442/33442_test_max_less_equal.c" title="test_max_less_equal.c attached to LU-8465">test_max_less_equal.c^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> file tests that min_larger_equal() and max_less_equal() is correct.</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36006" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36006</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: debug e2fsck_insert_dir_info<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: b4d83091ca90ad1a8acb58cb03a2ffdd093b612c</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36018" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36018</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge dir_info after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: f67bccf841a002ff06f3fc2d5a9df54dd1a21c2e</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36019" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36019</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: free bitmaps even thread quit<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: c6b9ad0533ec66b75c62390eccd5043f5773267a</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36020" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36020</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: rbtree bitmap for dir<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 4efa96732f0c9ba625caee91a1e9e8c320b770c1</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36026" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36026</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge badblocks after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: c5214a132138451f90f5d5ce966df02ce534ed7d</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36044" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36044</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge inode_link_info after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 46db7ec72a7e5fd5160cf65e798f0a9f1b3a44ed</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36053" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36053</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge dblist after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 37b1bc16ea047951910c39574ff757d9c4431379</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36054" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36054</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add debug codes for multiple threds<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: a7a62d8286c833a6b04cc3d356465fe8a448067c</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36079" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36079</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: do not start thread without any work<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 44ea3c360935f4bd5c19bc5842181cedb7662769</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36080" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36080</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fixed a bug of rb_merge_bmap<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 30f7489acaceb27f4e36a1414712962d3223e7d5</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36081" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36081</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: only reserve block for root in the first thread<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: dcf826973e7e07369ac5e8b9f942bfe099de4965</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36083" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36083</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix problem of pthread debug<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 984d49cc207e2f2d617ce4ca58daf47e15291cbb</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36084" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36084</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: clear the reserved blocks when restart<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: b89f249ba2e150a4dc2906ba48a486e21fd9e6ad</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36097" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36097</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge counts when threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 4f1c0aa71ff514a0d131c3facb59eb54228a7188</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36098" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36098</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge fs flags when threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: b7334aea79e73ec11e17bf06b55add9f36d9096d</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36099" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36099</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge icounts after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 1f15801a512ac3d3bf671f5ec162d14b964b85f1</p> <p>Looked at current patches, i think it might have a design problem that was ignored, e2fsck works as following:</p> <p>e2fsck_pass1-&gt;e2fsck_pass1e-&gt;e2fsck_pass2-&gt;e2fsck_pass3-&gt;e2fsck_pass4-&gt;e2fsck_pass5</p> <p>And existed patches flow was trying to use pthreads in e2fsck_pass1(), then merge results after e2fsck_pass1 and continue.<br/> The problem is there isn't any memory sharing module to make sure how this could be worked?</p> <p>As in the threads of e2fsck_pass1() we will malloc memory which will be used later, the current model as i understand is:</p> <p>Parent process-&gt;pthread_create sub thread-&gt;sub thread malloc memory-&gt;Parent process try to access sub thread memory?</p> <p>I don't think this will work? We might need some good memory sharing frame to work this out?</p> <p>&gt; The problem is there isn't any memory sharing module to make sure how this could be worked?</p> <p>Shilong, there will be certainly some need to sync between different threads in some circumstances. For example, bitmap is updated by a thread, that means the bitmap would need to synced to all of the other threads too. That part is not finished by the patches yet.</p> <p>In order to sync between threads, we might need to change the logic of the e2fsck_pass1() a little bit to have some sync step before trying to fix the problem. There are other ways, I am not sure which one is better though.</p> <p>&gt; Parent process-&gt;pthread_create sub thread-&gt;sub thread malloc memory-&gt;Parent process try to access sub thread memory</p> <p>No, all the memory allocated by the sub threads will be merged to the global thread, and then freed.</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36862" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36862</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix to return ENOMEM in alloc_size_dir()<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 7ece6455f4794a967cbc6313de7760707d23fbf0</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36863" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36863</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix use after free in calculate_tree()<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: c98eea9615b7212d816087b62778ff1588d314ea</p> <p>I was reading the <tt>pthread(7)</tt> man page, and it states:</p> <blockquote> <p> POSIX.1 specifies a set of interfaces (functions, header files) for<br/> threaded programming commonly known as POSIX threads, or Pthreads. A<br/> single process can contain multiple threads, all of which are executing<br/> the same program. <b>These threads share the same global memory</b> (data and<br/> heap segments), but each thread has its own stack (automatic variables).</p></blockquote> <p>So we <em>should</em> be able to have e.g. a shared block and inode bitmaps as long as they are allocated before the threads are forked (so they all know where the bitmaps are allocated), and we use locking to avoid multiple threads updating the same structures at the same time (words or rbtree).</p> <p>We can use a scalable <tt>pthread_mutex()</tt> implementation for updating the block bitmaps (e.g. block group number hashed across 4*num_threads locks, similar to <tt>struct blockgroup_lock *s_blockgroup_lock</tt> in the kernel). The inode bitmap should be relatively uncontended, since we will be splitting whole block groups between threads, but it should still use a scalable <tt>pthread_mutex_lock()</tt> for access like with the block bitmap. I think it makes sense to lock whole groups rather than individual words, since it is likely extents will span hundreds of blocks, and threads will be processing thousands of inodes in the same bitmap.</p> <p>I <em>think</em> there are enough abstraction hooks in libext2fs for the bitmap handling that we can register our own callbacks from the code to avoid embedding pthreads into the core library (which will otherwise cause problems for non-pthread users). It may be a bit tricky to consolidate the rbtree bitmap with parallel access, but we could potentially lock the nodes of the rbtree similar to htree lock instead of the groups?</p> <p>Note, in addition to the bitmaps, it probably makes sense to push the problem handling code to the main thread. This could be done with a producer-consumer model. While single-threading the error handling is not as fast as doing it in parallel, I think it may too complex to decide which problems can be fixed by a thread (and in parallel). It <em>may</em> be that all pass1 errors can be handled immediately (i.e. they only affect the current inode and mark its blocks set in the bitmap), but this would need to be verified.</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37022" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37022</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: Don't use EXT2_FLAG_EXCLUSIVE for multiple threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: efc91105a42c60fad824db4a6b0759247628b3b0</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37024" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37024</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge dx_dir_info<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 244851fbb6d644a00977a29e4a7a4cd6548b499a</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37023" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37023</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix wrong memory access in ext2fs_merge_dblist()<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 95939129d422dd6cb1547a6aa8fc0ff69104a82a</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37026" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37026</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: split and merge invalid bitmaps<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 76b103f3c503554813374a040165776c074098e1</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37025" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37025</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: make threads splitting aware of flex_bg<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 4a129c8c4e9616cb1e29da503f524405c19e5ff1</p> <p>Since (AFAIK) the block and inode bitmap code is using an rbtree, I think there are two options for how to handle these bitmaps between threads:</p> <ul class="alternate" type="square"> <li>a single global bitmap that has locking to protect all thread access (the locking itself could be hashed to avoid the majority of contention, but potentially every bit modified would need to get a lock). The benefit of this model is that it is easier to switch between threaded and serialized operation.</li> <li>each thread has a separate bitmap and there is a function to merge bitmaps at the end of pass1. The drawback of this approach is that the bitmap is not visible to other threads until they have finished their work, which <em>may</em> prevent pass1 repair actions, but maybe they do not depend on these bitmaps to do anything? Possibly inode badness depends on a global badness value, but this is unlikely to be a performance bottleneck since it is rarely updated.</li> </ul> <p>It would probably be worthwhile to run a quick benchmark to call <tt>pthread_mutex()</tt> in a loop on an uncontended lock to see how many locks/sec it can get,then run the same for two threads on the same lock to see what the aggregate locks/dev they can get, and two threads locking different locks. This will help make the decision whether the locking is expensive and should be avoided at all costs, or whether lock hashing would be enough to avoid contention in most uses. </p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37657" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37657</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge group desc<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 4df9839097114c3b91429f7bae8fe5767a457270</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37658" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37658</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: only allow first thread to recreate resize inode<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: d9ec4dc8f8644cba37e6846b6ca1ab49d9344a64</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37782" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37782</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge dirs_to_hash when threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: dd2f378a63c9bbadc7b462ddf53f14489068b817</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37784" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37784</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix to avoid splitting too much for small fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: da3471ebb572ee54ae4ea0a072c00df4ec70682b</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37783" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37783</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge context flags properly<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: c6a049258fa23fa153073a8e04bad0f77f8dec42</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37785" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37785</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix to use correct list in merge_u32_list()<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: edda4004d9885c760eb0a51328c8c508e7c5cfcd</p> <p>Currently, i've tested the whole series patches with number of thread 8, and all test passed with e2fsprogs, but there are still some problems that i want to discuss here:</p> <p>actually, for pass1, parallel repair is possible, as mostly of repiar for it is just inode rewrite(flags, size etc), but there are still some possible issues to be global:</p> <p>1) found_block, dup_blocks.<br/> This is used to fix duplicated blocks, this need be global between different threads as scanning going on, so that it could find duplicated blocks correctly, otherwise, after merging finish, it just mean no duplicated blocks within every thread context, we still need find every blocks to check which might make things complex.</p> <p>2) block allocations/free,<br/> This doesn't happen much, but still possible for pass1, etc recreate resize inode, relocate bitmaps etc, i think we shall make this operation exclusive for each threads.</p> <p>3)superblock<br/> This is global information, it is nature to make it global rather than merge it after threads finish, especially, there are a lot of areas inside it.</p> <p>So potentially, there could be two ways to address issues like this:<br/> 1) We only push all fixes to master thread(let's say it as thread 0), all other threads just try to do reaonly open and try readonly check once it find some errors, push it to master thread and let master thread do real fix.</p> <p>Good benefits of this is:<br/> 1.1 we could keep EXCLUSIVE open flag for e2fsck.<br/> 1.2 it looks safer to guarantee correctness, since only one writer.</p> <p>Bad side is:<br/> 1.3 more modifications to core codes of pass1, and need some proper memory sharing model to guarantee correctness, and there might be some gap from finding problem and fixing it, might break some depedency for this though.</p> <p>2) Allow parallel fix of pass1(what current series did)</p> <p>Good benefits of this:<br/> 2.1) Less changes to core codes which might make it less error-prone.<br/> 2.2) real parallel fix which is faster.</p> <p>Bad sides.<br/> 2.3) Break EXCLUSIVE open of e2fsck if we try multiple threads.<br/> 2.4) need different mutex/read/write locks to protect global variables, and serialize some criticial fixes between different threads.</p> <p>Any ideas or input what is better idea that we could go?</p> <p>Comparing to parallel checking, parallel fixing isn't so important, because 1) most file systems are not broken and 2) most part of a broken file system are not broken 3) it is more acceptable to fixing a broken file system more slowly but less risky than more quickly but more risky.</p> <p>So, if parallel fixing is not super important, I think the top challenge here is how to write trustable fix codes. So, if putting all fixing to a single thread is simpler and clearer, then we should do that. But I agree with you that this is not going to be the case. Each thread has its own checking and fixing process. Fixing can only happen naturely when checking finds a problem. I am not sure it is easy to put all codes into a single thread.</p> <p>And fixing the problems in parallel by multiple threads look dangerous too. The reason is, essentailly multiple threads write to the same shared device. And if locks are not protecting the ciritical region correctly, data will be corrupted.</p> <p>So, instead of either the upper two ways, I think maybe the following ways is better. Each thread do the check and fix by itself. But whenever the thread need to change/write anything (either in memory or to disk) that could be shared by other threads, it should do that exclusively in a special fixing context. Before the changing/fixing, all threads might need to sync with the fixing thread on the shared information (block bitmap etc). And after the changing/fixing, all threads might need to sync the fixing thread again so everyone agree on the change. And during the changing/fixing, no other thread do anything including doing I/O, so that the fixing thread can do I/O exclusively or change anything it wants.</p> <p>I think we can start by strict policy first, i.e. anything that we are not sure whether is shared by others, we can use the fixing context to protect it. When we know more about the logic details and performance bottlenecks, we can future use other ways to remove possible bottlenecks. And in the end, the parallel fixing process will be turstable, stable and at the same time quick.</p> <p>Comparing to fixing the problem in a dedicated thread, we don't need to change the current fixing codes a lot. We just need to add codes to enter and quit the context.</p> <p>Comparing to adding a lot of mutex/read/write locks, this doesn't require deep understanding of all details. And we are pretty sure the fixing and modification won't cause any data corruption due to race or overlapping.</p> <p>What do you think?</p> <p>All threads might need agreed changes before going on, this might be problem, actually this is exclusive operation:</p> <p>Considering superblock cases, we have 100 threads, eveytime, we need dirty superblock, we need copy 99 * 4096 buffer head, maybe just one mutex to hold on whenever fix is needed for possible shared contents modified is simper, in this case, thread contexts<br/> need share memory from global context for some cases.</p> <p>I agree with Li Xi, that the speed of fixing problems is much less critical than the scanning of correct parts of the filesystem, and the correct fixing of errors. The only main slowdown in pass1 is the duplicate blocks handling case (pass1b/c/d), and I think we should generally try to avoid that (adding/improving the "inode badness" patch to better catch cases where one inode is a duplicate of another).</p> <p>The <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/34393/34393_vault20-pFSCK-domingo.pdf" title="vault20-pFSCK-domingo.pdf attached to LU-8465">vault20-pFSCK-domingo.pdf^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> code took the approach where the "fix" was done in the context of each thread, but other threads were blocked during the fixing process. This serialized the threads and prevented parallel fixing of individual problems, but each thread could be scanning in parallel. That said, it wasn't clear what the memory model is in their pFSCK (globally shared or per-thread). That might imply that a mutex covering the "<tt>Fix? y</tt>" case would be enough to block out other threads from repairing the filesystem concurrently, but it wouldn't prevent them from changing other parts of the filesystem (e.g. bitmaps, arrays) that might be needed by the thread doing the repair.</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37825" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37825</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: split and merge quota context<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: edbed5c9bce76994735f8e1a5360b9986db8b4c3</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37826" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37826</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: serialize fix operations<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: e35249ed353ef7d9f1b712de8847415e4a6376c9</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37854" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37854</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: share one global superblock<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 4051baea11a3eca68a860b3898d19d53c37bc5fd</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37855" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37855</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: share global group desc<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 8834c3d449530f4e3f40b6ef5fe81db4576105d2</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37856" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37856</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: move some fixes out of parallel pthreads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 1464d92d2a346942c5da567348cb015513e2891e</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37857" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37857</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix to protect EA checking<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 617a0a257c171314e87b76ddc436694954a7b70a</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37858" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37858</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix to merge quota updates<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 1bc8b606300ee2ec669c5a20a686825c19001667</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37859" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37859</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: allow admin specify number of threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 41326f0d7d10e5aab05f8dd45c9c781d7cd46209</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37885" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37885</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix to calculate invalid_bitmaps<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: eeca027bed3bb2fb4f297303b02cf9b08fdda772</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37887" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37887</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix readahead for pfsck of pass1<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: bfbebd4bc5a811ca3b9c132de64a661c441de5c7</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37886" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37886</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix to get_lost_and_found before threads start<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 77ead2e939c2de1c1051fdba313259a1ac18592e</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37884" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37884</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: kickoff mutex lock for block found map<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 7f0b446dbbac18768c677b518282212ed4073cf3</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37905" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37905</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: kick off ea mutex lock from pfsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 86ef85e6f2e8a5f39209227c92efbf675fd1ac70</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37995" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37995</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge encrypted dir lists after thread finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 451649cf5292859ee4f3061798a801946b2adec5</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37999" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37999</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: reset lost_and_found after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: b527050b055605273c7a2a467df98cfa3bf7c1ac</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37997" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37997</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge min_extra_isize after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 57f077a370db8c9b20566cf8db8af4c187f59593</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37996" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37996</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: simplify e2fsck context merging codes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 8b193d2dd65d767699414d33d7ff252c72814913</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37998" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37998</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge options after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 52f6d7e5428bf25ba1390246aba0e502ed590edd</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38000" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38000</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge extent depth count after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 6b81e80b90bf20450348b79f5e07ee53d9f60e20</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39840" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39840</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: remove unused fs_ext_attr_inodes/blocks<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 1a8d5b742965e45d428fa605ad3ea8e35f009f37</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39841" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39841</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge icounts after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: d2660c3a05b331fda3ea143dda50b9a9adb0dde5</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39843" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39843</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge inode_badness after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 7ba7c8aa34182777e5482b5d1e95daa995869e17</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39845" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39845</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: only setup threads if -m option required<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 9bfbd37b98fc5b0bb65c9962135ca6a50c96a5d6</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39844" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39844</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: set E2F_FLAG_ALLOC_OK after threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 9ef377d98d85a717485fe7189472d0777832d9b5</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39842" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39842</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge inode_bad_map after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: f82ee5ab3e7eee9b289245dac1fcdf28ab591a3d</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39846" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39846</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: wait fix thread finish before checking<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: c8ecbc8f5ebb83d891366a4d027aea7216c2c13a</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39847" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39847</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix merge dblist and badblocks properly<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: fdee28ee36c45284a47e174570e11ed7c5976520</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39848" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39848</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix to free icache leak<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 8d83b8c210b118a48b8fab34f2def625b07776d8</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39850" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39850</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> ext2fs: fix to avoid invalid memory access<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 7831f11762c38d358644022c67bfffef1e794264</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39849" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39849</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: avoid too much memory allocation for pfsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 0ade56d171a0c5a4b487d8225dff6a4992375fb8</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39851" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39851</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> ext2fs: parallel bitmap loading<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: c91e78e9ea9bf6248c1f3434e1c31ed5987888f0</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39852" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39852</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: adjust number of threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: e957e7c1903a5b80af133d8fbe9cce573abe48b3</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39874" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39874</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: skip mmp update for multiple threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 5e6e07ce8c7efaf0f4223260e8e49ecc4f4f057f</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39914" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39914</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: make default smallest RA size to 1M<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: d1df7948228334274d0dd5f86d0fc5e2aae8ab8d</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35684/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35684/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: cleanup struct e2fsck_struct<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 8ba5db339a5fbf8e2dbde1b517879c0a7de6f5bb</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39840/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39840/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: remove unused fs_ext_attr_inodes/blocks<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 828f7975917c5351ba2616ae28aa737857b605a0</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35689/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35689/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add -m option for multithread<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: e80445181c12f55a61a6e4d4b178d2960d0e403d</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35690/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35690/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy context when using multi-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 59c8379f2b51885ee74b248ab93bdf742c1eb96d</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35696/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35696/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy fs when using multi-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 7432f421b844b5cc6c84db05454d7cfdba1d449f</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35698/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35698/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy dblist when using multi-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: e5058d7508eec4997fb0a0858d4ad80ee9206798</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35701/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35701/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: clear icache when using multi-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: d8c6866b683bf0007ef59306ff667cd3de57fbe8</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35710/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35710/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add assert when copying context<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: cbf3c4205eb174b2141d2b160557823b89632d40</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40014" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40014</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: clear icache when using multi-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 542a0eeb8f44d9f0a0ceef7a40f3a052a527f86c</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40015" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40015</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add assert when copying context<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: e1acbb00bb1c8635ed0626dbf771b28b8449ce36</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40017" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40017</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> test: add pfsck test<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 8f12d37566dec681ebda800a9f5ea417190b6db9</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40021" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40021</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> configure: enable pfsck by default<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 592c95b285837cf30b0c16780a6fbe6aa9ebac5a</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40024" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40024</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: reset @inodes_to_rebuild if restart<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 276cff391901e38601728cab11f7638d3732b4b2</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40014/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40014/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: clear icache when using multi-thread fsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 321ed5ab1d11ca2e74adbd84222f96a6dc180da9</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40015/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40015/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add assert when copying context<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 18a7cf47f057c152ba76f5bf82b0ff4504ab116a</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35712/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35712/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy bitmaps when copying context<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 87444962ca6ef43f3f2fca6a5f8c3750961f1237</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35726/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35726/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: copy badblocks when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 1d9178988c187efa610c97c36ee0d6c3d410f947</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35763/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35763/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: open io-channel when copying fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 0db312fbbc5d6a7b6cd5fa43b513e9fdfd9a717c</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35780/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35780/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: create logs for mult-threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 2023afea8aa825da45de7a66cee9e8182dd79cf9</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35793/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35793/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: optionally configure one pfsck thread<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 727154449dd68c7b3197d8cc4d0558f3826dc692</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35846/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35846/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add start/end group for thread<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: b2a5855151444140949e8e2ec82fe01b802ada41</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35855/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35855/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: split groups to different threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 0c20759c1cdf00c6436115842a36ad1e094583f3</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35922/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35922/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: print thread log properly<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 6a78e90e76bd259b8df436d0bbfd19b411fde94b</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35961/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35961/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge bitmaps after thread completes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: fba469b0ef2449bed72d2fb91c9e4a0374ea71ef</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36001/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36001/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: do not change global variables<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 1f6e8b81c2d964f54c6615ff775a3f6c3a9bbe1f</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36004/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36004/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: optimize the inserting of dir_info_db<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 867621ddf45ab4c8b59bdb45998184b0d6ea411d</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36018/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36018/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge dir_info after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 70127eb62062f6b07b4209aefa3346c354bd9afd</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36020/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36020/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: rbtree bitmap for dir<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: ac7a05ae0150880530483c41fe108f1bb8cc5908</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36026/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36026/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge badblocks after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: a16f8183604fbb8de18fbf5221502cd86b25ac0c</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39841/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39841/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge icounts after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: d28c91699d81615aec6a4c5b969d307f96af775d</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39843/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39843/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge inode_badness after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 1054ce57781b9f82fd749a4916529aed4b093d52</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36044/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36044/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge dblist after thread finishes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: bc9f0ed402d5b4ac01a84f476e94ef07dc452604</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36054/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36054/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: add debug codes for multiple threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 6b4895ca7e6fbe1b130e3603a5edde6a1570f453</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36097/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36097/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge counts after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 2c1fdf10283ad3d28d51717ef10eacbff4e28d25</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36098/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36098/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge fs flags when threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: e51f6bfe6d699e49cb8f8c8f3ae15db334077ac3</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37024/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37024/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge dx_dir_info after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: fddefccc5ac0c1311e5371ae834c1dc20aba8133</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37782/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37782/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge dirs_to_hash when threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 27de25461f5fb0d2cc40383f5219a9dcfd219245</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37783/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37783/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge context flags properly<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 0f87b70e33d21ca72ac554ab2cb4a0f4df9e3231</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37825/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37825/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge quota context after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 0edcac8b4a63d8f469c55f9a0c1842068c98eb08</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37826/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37826/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: serialize fix operations<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: aa67a29fe0ae62f6f15586e0ad159a12786bd252</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37856/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37856/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: move some fixes out of parallel pthreads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 1f55c479ba55fb95bab0cc564b7568a19e8e67d8</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37885/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37885/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: split and merge invalid bitmaps<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: ee70eff3f282a1b6500df5c895affd155811c7af</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37905/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37905/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge EA blocks properly<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 9ec0c7091c2c1d4a031c3e899d65c8ac1ce85751</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37884/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37884/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: kickoff mutex lock for block found map<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 613da4b91de3c482ffd33cfe6ceb345d402c21ee</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37859/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37859/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: allow admin specify number of threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 62a0e774c26f451769df3685298d8e183f2a1bba</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39852/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39852/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: adjust number of threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 1b614e6fbd38320557bc904d567ba1046b566118</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37887/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37887/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix readahead for pfsck of pass1<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 15077733b979ba2fc990d21a5457246bed8d1fc9</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37995/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37995/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge encrypted dir lists after thread finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 18538b2721c8fc6072a8892a439f5ab9a24692db</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37997/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37997/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge expand_eisize_map properly<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: cb4f0f44147a1fe3fde5b56126dce10b150aa5ba</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37998/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37998/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge options after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 161ab07533b70349de98b889ac1c9b3b38145296</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37999/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37999/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: reset lost_and_found after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 4ee6ff216e0a4898e55280ceb274f8f06915e2fd</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38000/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38000/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge extent depth count after threads finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 8ce4b41a3786dfcce6d02f1e4faf3dcbd62c7a46</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37996/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37996/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: simplify e2fsck context merging codes<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 4e9d8e66a30ad786111788fd50b8d95785f81555</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39844/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39844/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: set E2F_FLAG_ALLOC_OK after threads<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 5a959732ebfd301b6f08ef3dba22cd3cd68d4755</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39846/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39846/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: wait fix thread finish before checking<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: f98ba79539969b3b01d927fbb4d0bd6fa4adeb39</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40016/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40016/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: cleanup e2fsck_pass1_thread_join()<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 59b7eac3b747afa1027cd9a905ed08d5fd8c3d54</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39849/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39849/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: avoid too much memory allocation for pfsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: a6a2410f4fa0b02f3fda8876d4b6037ddc6ef65c</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39850/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39850/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> ext2fs: fix to avoid invalid memory access<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: d8b23ec2a3a900ecf2040ea67a9b4b34d499c536</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39914/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39914/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: make default smallest RA size to 1M<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 38247c218eed14f510953906e406352c4c9a6126</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39851/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39851/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> ext2fs: parallel bitmap loading<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 0e888c3cca27332cfc46e165a5b726c4f0681f00</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39874/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39874/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: update mmp block in one thread<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 7284939c5761844ef0da56b4853650010683f828</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40024/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40024/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: reset @inodes_to_rebuild if restart<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 13848abab6290fd7bba77527ec9289a5e3814835</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40060" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40060</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix build for make rpm<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 5bc0acc684d4732b064c3dae73b1754405f5f5a7</p> <p>Attached is test results of pfsck and compared with fsck today. <br/> increased readahead improved at pass1 and downed elapsed time from 7700sec to 2100sec even single thread.<br/> parallel scanning speedup at pass1 and pass5. In the end, the elapsed time downed from 19,700sec to 2100sec(64 threads) and 1780sec (256 threads) which is more than 10x speedup!<br/> "-m 2" didn't scale in this case, because only 40% inodes (400M inodes) were populated, thread0 covered range <span class="error">&#91;0, 4730880&#93;</span>, but thread1 covered for range <span class="error">&#91;4730880,9461760&#93;</span> which was almost free. So, only a thread was busy and it was similar speed with single thread.</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40063" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40063</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix race in ext2fs_read_bitmaps()<br/> Project: tools/e2fsprogs<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c5155aed04fa7d843c0766ff43458404ee3b3b13</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40060/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40060/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix build for make rpm<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: b6c4cadb62d25e91aeebcb619979a9cba6887ebb</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40021/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40021/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> configure: enable pfsck by default<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 34556b33f6d723c6960e055ee2916c43a1c1e3f2</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40017/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40017/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> test: add pfsck test<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 475a417008e3724a5b7190a5854f197911754ab6</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40065" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40065</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix race in ext2fs_read_bitmaps()<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 231f0a4117c4c7213b2fdbed2ecd86cc365dbd2a</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40066" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40066</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix readahead for pass1 without pfsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 6c2612804b839e4e88d74fde4c90ba3b60782157</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40068" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40068</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix f_multithread_ok test<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 978cdf5597ce65577eea70fd4a05d4edbb682260</p> <p>Wang Shilong (wshilong@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40068/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40068/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix f_multithread_ok test<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 2b281447e28f7cf0bfe3950b427b33e797b6bb64</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40069" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40069</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix memory leaks with pfsck enabled<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 2b85e98c790a0cca7523076fc51f860fd443ae0d</p> <p>Wang Shilong (wshilong@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40065/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40065/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix race in ext2fs_read_bitmaps()<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 3b4ff5f207054b6f5507a986ad8522a08c2d8228</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40070" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40070</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> tests: enable pfsck if possible<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9f98a1b776134d65f054f9e073298c4f467c67f9</p> <p>Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40071" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40071</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> ext2fs: fix to set tail flags with pfsck enabled<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 104479260a753dc021806fbc003865a9804bc3a6</p> <p>Wang Shilong (wshilong@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40066/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40066/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix readahead for pass1 without pfsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: b4d8cf38c59cd4e85d3816368a2434891e16e84d</p> <p>Wang Shilong (wshilong@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40069/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40069/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: fix memory leaks with pfsck enabled<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 8d561b5d93a00d3b227917a0ff25da5fd5d64ffc</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40075" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40075</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: misc cleanups for pfsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 98c0a9b4f8a02753a1d789fc4da0bca5150a23b9</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40071/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40071/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> ext2fs: fix to set tail flags with pfsck enabled<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 6505169f9a2a2cce0a8c383c1aa10292e01c6f4a</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40075/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40075/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: misc cleanups for pfsck<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: d9400e590b9f1ca026241501b22c7674d774da22</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40081" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40081</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> build: update version to 1.45.6-wc2<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: f2ddfd30f03c71e6ff7423ff816160e97a073baa</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40081/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40081/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> build: update version to 1.45.6-wc2<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 0b7bcc70a54425f53407fe7825caa4671b5bd082</p> <p>I think it fine to close this ticket, and we could open new ticket for further work.</p> <p>The e2fsprogs-1.45.6.wc2 build is available at <a href="https://downloads.whamcloud.com/public/e2fsprogs/1.45.6.wc2/" class="external-link" target="_blank" rel="nofollow noopener">https://downloads.whamcloud.com/public/e2fsprogs/1.45.6.wc2/</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40070/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40070/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> tests: enable pfsck if possible<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 55a4357424c5206910928f27f66db62465581150</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/43129/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43129/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> ext2fs: RO open for bitmap loading<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: c0f60f896182d26711c7d8d064dc95471b0d9829</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/44010" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44010</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge casefolded dir lists after thread finish<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 4aff8d51e30b20c4da49434a50ca49a104c0473e</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/44011" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44011</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8465" title="parallel e2fsck performance at scale" class="issue-link" data-issue-key="LU-8465"><del>LU-8465</del></a> e2fsck: merge expand_eisize_map properly<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 3055d6fa13f17f9794e3e8b7687ab69d59702bd8</p> Related LU-11882 LU-12051 LU-7370 LU-7592 LU-8974 LU-5949 LU-14126 LU-14894 LU-14213 LU-14679 Development Rank 1|hzyj9j: Rank (Obsolete) 9223372036854775807