[#LU-8465] parallel e2fsck performance at scale

[LU-8465] parallel e2fsck performance at scale Created: 02/Aug/16 Updated: 26/Nov/21 Resolved: 30/Sep/20
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	None
Fix Version/s:	Lustre 2.14.0

Type:

Improvement

Priority:

Major

Reporter:

Artem Blagodarenko (Inactive)

Assignee:

Wang Shilong (Inactive)

Resolution:

Fixed

Votes:

Labels:

None

Attachments:

128_threads.txt

pfsck-test.xlsx

test_max_less_equal.c

vault20-pFSCK-domingo.pdf

Issue Links:

Related
is related to	~~LU-11882~~	OST recreated objects gets badness ma...	Closed
is related to	LU-12051	ldiskfs directory shrink	Open
is related to	~~LU-7370~~	improve e2fsck lost+found insert perf...	Resolved
is related to	~~LU-7592~~	Сhange force_over_128tb lustre mount ...	Resolved
is related to	~~LU-8974~~	Сhange force_over_256tb lustre mount ...	Resolved
is related to	~~LU-5949~~	Verify e2fsck inode badness is workin...	Resolved
is related to	~~LU-14126~~	parallel e2fsck does not work well wi...	Resolved
is related to	LU-14894	Parallel pass2 support for e2fsck	Open
is related to	LU-14213	enable parallel e2fsck by default	Open
is related to	~~LU-14679~~	parallel e2fsck for pass2 directory s...	Resolved

Rank (Obsolete):

9223372036854775807

Description

Mentioned in ~~LU-7592~~ by Andreas Dilger

e2fsck performance will become an issue at this scale, and it would likely need to be parallelized to be able to complete in a reasonable time. It could reasonably expect multiple disks at this scale, so having larger numbers of IOs in flight would help, as would an event-driven model with aio that generates lists of blocks to check (itable blocks first), submits them to disk, and then processes them as they are read, generating more blocks to read (more itable blocks, indirect/index/xattr/directory blocks, etc), repeat.

Comments

Comment by Andreas Dilger [ 08/Dec/17 ]

I think that before you start looking at format changes to improve e2fsck,
that you look at what can be done to improve e2fsck performance on existing
filesystems. I think there is a lot that can be done, especially with SSD
or RAID-10 storage that is mostly idle during the scan. This has the benefit
that it helps all filesystems, instead of only new ones, and improvements
that do not change the disk format are more easily merged.

There was work done in 1.42 or 1.43 to do async readahead of inode tables
and other blocks, which improved e2fsck performance, but I suspect there
is more that could be done in this area.

Firstly, an example e2fsck run on my home OST and MDT filesystem:

myth-OST0000: clean, 359496/947456 files, 781092161/970194944 blocks
e2fsck 1.42.13.wc5 (15-Apr-2016)
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 5116k/932k (4903k/214k), time: 181.93/ 4.12/ 1.84
Pass 1: I/O read: 160MB, write: 0MB, rate: 0.88MB/s
Pass 2: Checking directory structure
Pass 2: Memory used: 11532k/1084k (4614k/6919k), time:  3.51/ 2.59/ 0.12
Pass 2: I/O read: 37MB, write: 0MB, rate: 10.53MB/s
Pass 3: Checking directory connectivity
Pass 3A: Memory used: 11532k/1084k (4617k/6916k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 11532k/1084k (4612k/6921k), time:  0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 225.43MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 5196k/932k (3932k/1265k), time:  0.09/ 0.09/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 7680k/932k (3917k/3764k), time: 227.05/ 1.72/ 3.03
Pass 5: I/O read: 187MB, write: 0MB, rate: 0.82MB/s

myth-MDT0000: clean, 1872215/3932160 files, 824210/3932160 blocks
e2fsck 1.42.13.wc5 (15-Apr-2016)
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 13092k/17500k (12876k/217k), time: 19.98/ 5.52/ 1.72
Pass 1: I/O read: 1104MB, write: 0MB, rate: 55.26MB/s
Pass 2: Checking directory structure
Pass 2: Memory used: 21012k/8468k (13732k/7281k), time: 23.28/ 7.54/ 2.25
Pass 2: I/O read: 1113MB, write: 0MB, rate: 47.81MB/s
Pass 3: Checking directory connectivity
Pass 3A: Memory used: 21624k/8468k (14212k/7413k), time:  0.00/ 0.01/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 21624k/5572k (13731k/7893k), time:  0.06/ 0.05/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 15.44MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 21624k/484k (2638k/18987k), time:  0.42/ 0.45/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 21624k/0k (2637k/18988k), time:  1.30/ 0.38/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 0.77MB/s

It shows that a large amount of time is spent in pass1 (inode/block scan).
The inode checking could be almost trivially multi-threaded, since each
inode is independent. It would need locking of shared structures. I
would make this an event-driven (producer/consumer) process, with threads
doing prefetch reads of inode tables, then processing the first group
on the shared list. This avoids threads sitting idle with a static
division of block groups if some groups are used more than others.

Pass2 (directory structure) is trivial on an OST because it has so few
directories, but is relatively longer on the MDT (which is SSD based).
During the pass 1 inode scan, a list of directory inodes will be generated,
which could start prefetching some directory leaf blocks to do leaf block
scanning. Again, this can be done largely in parallel as long as there is
reasonable locking on the bitmaps and a few other structures. It would
be good to have some sort of elevator for async directory block readahead,
to avoid random IO on HDD-based storage.

Pass 5 is slow on the OST filesystem because it is HDD based and is not
using flexbg, so there lots of seeking to read/update groups and bitmaps.
That probably doesn't need to be multi-threaded on any modern filesystem.

As for how to approach development, I'd start with a simple threading
model (pthreads) and add locking to the bitmap and list structures (maybe
a 64-slot hashed lock for bitmaps like include/linux/blockgroup_lock.h in
the kernel, instead of one lock per bitmap block, since most are unused).
For ease of acceptance, it should be possible to disable threading and
locking at compile time for single-threaded use (e.g. embedded or non-linux).

Then, a producer/consumer model for processing inode table blocks could
just be split on a per-group basis with very little change to the code.
Even just a simple counter for "last processed group" to track which
group an idle thread should work on next might be enough. These threads
would also submit async inode table readahead of maybe 3*num_threads
(about 1GB/1M inodes) to ensure threads are not waiting on disk IO.

Finally, a separate producer/consumer model for processing directories
that are found during pass1 could be added. I'm not sure, without
checking the code, whether pass 2 can be started before pass1 is done.

With the addition of larger directory support, this would benefit from
being able to shrink ext4 directories online, so that directories shrink
as old entries are removed. There have been many discussions on how to
implement ext4 directory shrinking, so let me know if this is something
you are interested in.

Cheers, Andreas

Comment by Andreas Dilger [ 19/Mar/19 ]

Artem, I know this is something you were interested in previously, have you done any work in this area?

Comment by Artem Blagodarenko (Inactive) [ 22/Mar/19 ]

Andreas, not so far we faced with e2fsck slow work on one of our system. And finally make ~~LU-11882~~ patch. All other work is planned, but currently is suspended.

Comment by Li Xi [ 02/Aug/19 ]

Patch: https://review.whamcloud.com/#/c/35597/ cleanup e2fsck_pass1

Comment by Gerrit Updater [ 02/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35671
Subject: ~~LU-8465~~ e2fsck: cleanup abort in e2fsck_pass1_check_inode
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: f26dd61c2b2aad9cd7cda0df198fe3c8b7eed2a8

Comment by Gerrit Updater [ 02/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35676
Subject: ~~LU-8465~~ e2fsck: cleanup bad block in inode table in e2fsck_pass1_check_inode()
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 53959bf250cc036e77fc54b9417d04085839a1d1

Comment by Gerrit Updater [ 02/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35678
Subject: ~~LU-8465~~ e2fsck: add ES_RESTART for enum e2fsck_status
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 37f60cc06b759172b5850dea5df55d9d47b2eb5d

Comment by Gerrit Updater [ 03/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35684
Subject: ~~LU-8465~~ e2fsck: cleanup struct e2fsck_struct
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: ea099b12438dd826b9719d8493287367370dd664

Comment by Gerrit Updater [ 04/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35689
Subject: ~~LU-8465~~ e2fsck: add -m option for multithread
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: a458233a5dc11979dd08b8c74dc7785eb6dfc58a

Comment by Gerrit Updater [ 05/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35690
Subject: ~~LU-8465~~ e2fsck: copy context when using muti-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 16d2387e62a0f7b8c2e296f76f39590d57b0a537

Comment by Gerrit Updater [ 05/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35691
Subject: ~~LU-8465~~ e2fsck: debug
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: bd6215e5adbec633006bdf5490ca99505867e237

Comment by Gerrit Updater [ 06/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35696
Subject: ~~LU-8465~~ e2fsck: copy fs when using muti-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 3b384dc9e0bf7b98433d65c7138e013982de3d5b

Comment by Gerrit Updater [ 06/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35697
Subject: ~~LU-8465~~ e2fsck: debug
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 7eff9d589624330db9815ad710703b11425f0f20

Comment by Gerrit Updater [ 06/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35698
Subject: ~~LU-8465~~ e2fsck: copy dblist when using muti-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: da7c00d0421667363b1bc5a70498efd1553b0b5f

Comment by Gerrit Updater [ 06/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35699
Subject: ~~LU-8465~~ e2fsck: debug "copy dblist when using muti-thread fsck"
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: abb9dbc99651d37c2d43e4a0a0757b96007e9327

Comment by Gerrit Updater [ 06/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35701
Subject: ~~LU-8465~~ e2fsck: clear icache when using muti-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 02d35235d637e44ac0a6feee623adfd971e05cf3

Comment by Gerrit Updater [ 06/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35702
Subject: ~~LU-8465~~ e2fsck: debug "clear icache when using muti-thread fsck"
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 34f00c6c4429600cd4fe1734e2eaf6c61a675085

Comment by Gerrit Updater [ 06/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35703
Subject: ~~LU-8465~~ e2fsck: debug "clear icache when using muti-thread fsck" without cleanup
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: df7a0daccf9a6cb7e1939ef89585790d9f3da02b

Comment by Gerrit Updater [ 07/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35710
Subject: ~~LU-8465~~ e2fsck: add assert when copying context
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 2d597965b393b5172208dd909b0c07658a7cd217

Comment by Gerrit Updater [ 07/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35712
Subject: ~~LU-8465~~ e2fsck: copy bitmaps when copying context
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: d5823b6137559d965162136bf281b8a0a834aae9

Comment by Gerrit Updater [ 07/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35713
Subject: ~~LU-8465~~ e2fsck: openfs rather than copying it
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 5bbaa98381535e853ac186daa18cd6da99194045

Comment by Li Xi [ 08/Aug/19 ]

The imeplementation of parallel fsck is going to very hard, not because of the new codes are hard, infact, new codes and logics are easy. The hardest part is how to add these codes without breaking any existing codes that exist and grow for decades. I am using several ways to confirm the patches don't break anything:

TEST #1. The regression tests under "tests" directory.

TEST #2. "valgrind" test to eliminate memory leak.

TEST #3. e2fsck on a huge Ext4 with hundreds of millions inodes to confirm no performance regression.

I am running all of these three tests to make sure every patch looks good. If you have any idea of other ways, please let me know and thank you so much!

The design is to copy the fsck context for each pass1 thread. And each context is seperated with other ones. When the fsck thread finishes, the global context will merge all contexts together, and continue run the pass2...5 tests.

And the plan is doing following steps one by one:

#1. Cleanup the codes if possible. Unfortunately, it is hard. According to my experience, it is so easy to break something. For example, https://review.whamcloud.com/#/c/35597/ passed all regression tests, but the fsck is slown down dramatically.

#2. Copy the context and its sub-fields to a new one, and run the same process like before. No thread will be created, and same functions are used even context is new one.

#3. Copy the context to multiple ones, and split the pass1 scanning of inodes to multiple steps. Each scanning step works on a range of inodes. And this will still be done without multi-thread. Each step uses different context, and these contexts will be merged into the global one. The global context will be used to run the pass2...5 check.

#4. Run all kinds of tests for step 3 by splitting the pass1 one scanning to different numbers of loops, from one to as large as the core number. This will make sure the context copying and merging process is correct.

#5. Create only one thread to run the pass1 check in a new context. This should still be able to pass all above tests including TEST #1. And it is impossible to have race problem of disk read/write.

#6. Create multiple threads to run the pass1 check in different threads. Each thread uses its own context, and works on a given range of inodes. This step is going to be very very hards because of races, including disk read/write conflict, global variable conflict and so on. So it will be necessary to split it into multiple steps:

#6.1 Roughly implement the multi-thread fsck and run it on a clean file system to make sure the checking works well. Performance numbers can be got after this step is finished.

#6.2 Implement the fully fsck and fix.

After step #6, TEST #1 isn't going to pass because the output of fsck is going to be changed. But still, we can change the fsck tests to:

FSCK TEST #1. Run fsck with multi-thread, but don't compare the result with the expected result of single thread.

FSCK TEST #2. Run fsck without multi-thread, compare the result with the expected second result.

All TEST #1 tests should be passed (maybe not for a few ones?). And this will make sure multi-thread fsck at least fixes the file system problems as expected.

Comment by Gerrit Updater [ 08/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35726
Subject: ~~LU-8465~~ e2fsck: copy badblocks when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 8e46169b9fa193f9ae9243a37eda824d4e0dcf08

Comment by Gerrit Updater [ 08/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35731
Subject: ~~LU-8465~~ e2fsck: copy mmap when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: cb0cb2a0402607afa6ae8bc343f6fbfe88793715

Comment by Gerrit Updater [ 09/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35748
Subject: ~~LU-8465~~ e2fsck: copy device name when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 46f170057b78417ebe009d91a41399ee47bf29e2

Comment by Gerrit Updater [ 09/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35753
Subject: ~~LU-8465~~ e2fsck: copy super when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4b660e2fcc160b1c507c692118d31af0c6cf0286

Comment by James A Simmons [ 09/Aug/19 ]

Is this being pushed upstream as well?

Comment by Li Xi [ 09/Aug/19 ]

Hi James, not yet. Will do when the performance numbers confirm the idea.

Comment by Gerrit Updater [ 10/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35763
Subject: ~~LU-8465~~ e2fsck: open io-channel when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: f2e83edde8df72a279951bbbb912fc7034a358ac

Comment by Gerrit Updater [ 12/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35770
Subject: ~~LU-8465~~ e2fsck: copy group_desc when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 2a5416da3a29337a294a6d2a988fc70881bbc1fb

Comment by Gerrit Updater [ 12/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35771
Subject: ~~LU-8465~~ e2fsck: copy image_header when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 6f49863cbe52d6b9f2a6453108b7e8b6ab1a2cc2

Comment by Gerrit Updater [ 12/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35772
Subject: ~~LU-8465~~ e2fsck: assert block_sha_map is NULL when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 3017502ad4f9bb2f176b4e029622f0dadb3fd64e

Comment by Gerrit Updater [ 13/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35780
Subject: ~~LU-8465~~ e2fsck: create logs for mult-threads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: ecb338d8acc7d54efb3c99432540b910bf6ef7cd

Comment by Gerrit Updater [ 15/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35793
Subject: ~~LU-8465~~ e2fsck: create one thread to fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 8c65f9f1b89a157f38ccf0c876c6abef9e532966

Comment by Gerrit Updater [ 21/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35846
Subject: ~~LU-8465~~ e2fsck: add start/end group for thread
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4618d1ee83d0c60ce51a5cf2f435f58b75a436ae

Comment by Gerrit Updater [ 22/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35855
Subject: ~~LU-8465~~ e2fsck: split groups to different threads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 820e9837cf6e5fc2a778be599c8deaf037f665a7

Comment by Gerrit Updater [ 26/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35922
Subject: ~~LU-8465~~ e2fsck: add perf debug info
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: d5a8739048a3956d80df3f2b2ca8ce36fcb31b99

Comment by Li Xi [ 27/Aug/19 ]

On a 1PB Ext4 file system, I created 105M inodes. And run e2fsck:

# df | grep sda
Filesystem                          1K-blocks     Used     Available Use% Mounted on
/dev/sda                        1099637648128 13563264 1088621733412   1% /mnt
# df -i | grep sda
Filesystem                          Inodes     IUsed     IFree IUse% Mounted on
/dev/sda                        1074397184 105887860 968509324   10% /mnt

With original e2fsck:

e2fsck 1.45.2.wc1 (27-May-2019)
MMP interval is 5 seconds and total wait time is 22 seconds. Please wait...
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 75116k/193852k (70544k/4573k), time: 3771.03/1472.81/31.63
Pass 1: I/O read: 52117MB, write: 0MB, rate: 13.82MB/s
Time used: 3771 seconds
Time used: 0 seconds
Pass 2: Checking directory structure
Pass 2: Memory used: 75116k/337408k (27172k/47945k), time: 98.10/86.34/ 3.37
Pass 2: I/O read: 2897MB, write: 0MB, rate: 29.53MB/s
Time used: 98 seconds
Pass 3: Checking directory connectivity
Peak memory: Memory used: 75116k/337408k (27172k/47945k), time: 3879.43/1568.92/35.05
Pass 3: Memory used: 75116k/337408k (25930k/49187k), time:  0.02/ 0.03/ 0.00
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Time used: 0 seconds
Pass 4: Checking reference counts
Pass 4: Memory used: 75116k/0k (21993k/53124k), time: 12.94/12.95/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Time used: 13 seconds
Pass 5: Checking group summary information
Pass 5: Memory used: 75116k/0k (20108k/55009k), time: 66.93/ 4.12/ 0.13
Pass 5: I/O read: 206MB, write: 0MB, rate: 3.08MB/s
Time used: 67 seconds

   105887860 inodes used (9.86%, out of 1074397184)
           1 non-contiguous file (0.0%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 105887853/1
   139657888 blocks used (0.05%, out of 275045679104)
           0 bad blocks
           1 large file

   105781892 regular files
      105959 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
   105887851 files
Memory used: 75116k/0k (20108k/55009k), time: 3959.37/1586.01/35.18
I/O read: 55235MB, write: 0MB, rate: 13.95MB/s

Using the e2fsck with patches https://review.whamcloud.com/#/c/35922/4

e2fsck 1.45.2.wc1 (27-May-2019)
MMP interval is 5 seconds and total wait time is 22 seconds. Please wait...
iii thread 0 is going to scan group [0, 32788)
[Thread 0] Pass 1: Checking inodes, blocks, and sizes
iii thread 1 is going to scan group [32788, 65576)
[Thread 1] Pass 1: Checking inodes, blocks, and sizes
iii thread 2 is going to scan group [65576, 98364)
[Thread 2] Pass 1: Checking inodes, blocks, and sizes
iii thread 3 is going to scan group [98364, 131152)
[Thread 3] Pass 1: Checking inodes, blocks, and sizes
iii thread 4 is going to scan group [131152, 163940)
[Thread 4] Pass 1: Checking inodes, blocks, and sizes
iii thread 5 is going to scan group [163940, 196728)
[Thread 5] Pass 1: Checking inodes, blocks, and sizes
iii thread 6 is going to scan group [196728, 229516)
[Thread 6] Pass 1: Checking inodes, blocks, and sizes
iii thread 7 is going to scan group [229516, 262304)
[Thread 7] Pass 1: Checking inodes, blocks, and sizes
[Thread 4] Pass 1: Memory used: 205208k/1349632k (204375k/834k), time: 1407.46/11259.11/ 0.78
[Thread 4] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s
[Thread 6] Pass 1: Memory used: 205276k/1349632k (203804k/1473k), time: 1407.57/11260.05/ 0.63
[Thread 6] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s
[Thread 7] Pass 1: Memory used: 205308k/1349632k (203232k/2077k), time: 1407.66/11260.62/ 0.55
[Thread 7] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s
[Thread 3] Pass 1: Memory used: 205376k/1349632k (202660k/2717k), time: 1407.81/11261.19/ 0.82
[Thread 3] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s
[Thread 1] Pass 1: Memory used: 206752k/1349632k (202812k/3941k), time: 1408.34/11263.09/ 0.89
[Thread 1] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s
[Thread 2] Pass 1: Memory used: 206820k/1349632k (202244k/4577k), time: 1408.45/11263.34/ 0.86
[Thread 2] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s
[Thread 5] Pass 1: Memory used: 206892k/1349632k (201674k/5219k), time: 1408.51/11263.37/ 0.72
[Thread 5] Pass 1: I/O read: 1MB, write: 0MB, rate: 0.00MB/s
[Thread 0] Pass 1: Memory used: 254040k/1374780k (245271k/8770k), time: 3805.86/11436.56/10.64
[Thread 0] Pass 1: I/O read: 52117MB, write: 0MB, rate: 13.69MB/s
iii 0 scaned group [0, 32788) 105887861 inodes
iii 1 scaned group [32788, 65576) 0 inodes
iii 2 scaned group [65576, 98364) 0 inodes
iii 3 scaned group [98364, 131152) 0 inodes
iii 4 scaned group [131152, 163940) 0 inodes
iii 5 scaned group [163940, 196728) 0 inodes
iii 6 scaned group [196728, 229516) 0 inodes
iii 7 scaned group [229516, 262304) 0 inodes
Pass 2: Checking directory structure
Pass 2: Memory used: 254040k/1543484k (108778k/145263k), time:  0.10/ 0.03/ 0.08
Pass 2: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 254040k/1543484k (108778k/145263k), time: 3816.89/11446.48/10.78
...

Comment by Andreas Dilger [ 27/Aug/19 ]

In most filesystems the inode distribution is uneven. Groups at the start of the filesystem have more inodes allocated than at the end. In the absence of dynamic group allocation, it probably makes more sense to do interleaved group allocation, like one flexbg of inodes (128-256 groups) to each thread in round-robin order.

That can still be done without coordination between threads, but is more likely to load the threads evenly. It also is more likely to reduce seeking between threads.

It would be useful for your testing to capture blocktrace data and plot with seekwatcher to see the IO patterns of the parallel e2fsck, as it may be the parallel COU improvement is being offset by worse IO patterns (pass1 was previously perfectly linear IO).

Comment by Li Xi [ 28/Aug/19 ]

Thank you Andreas for the advices. We found the root cause of the long time costs for the threads that had actually no work to do because of empty block groups. The patch https://review.whamcloud.com/#/c/35659/ can fix the problem. I am going to check whether there is anyway to get a good number with that patch. And then optimize the group blance between threads as you suggested.

Comment by Andreas Dilger [ 28/Aug/19 ]

It isn't clear to me how patch 35659 ("libext2fs: optimize ext2fs_convert_subcluster_bitmap()") could fix this problem? AFAIK, that patch mostly relates to mke2fs.

Comment by Li Xi [ 28/Aug/19 ]

> It isn't clear to me how patch 35659 ("libext2fs: optimize ext2fs_convert_subcluster_bitmap()") could fix this problem? AFAIK, that patch mostly relates to mke2fs.

The pass1 check also calls ext2fs_convert_subcluster_bitmap(). The device has 1PB size, thus a lot of groups. But most of the groups are not used. ext2fs_convert_subcluster_bitmap() costs a lot of time. According to th e test result, each thread spends ~1400 seconds on this.

After using the patch, the time cost have been reduced to about 10 seconds.

Comment by Li Xi [ 28/Aug/19 ]

Following is a test log with patch https://review.whamcloud.com/#/c/35922/8

I used 128 threads to do the pass1 check. And only 13 threads have real work to do. It took at most 800 seconds for the slowest thread to finish.

The time cost of pass1 check would take 3771 seconds. That means, it might be possible to get speed up of 3771/800 = 4.7 times.

log: https://jira.whamcloud.com/secure/attachment/33433/128_threads.txt

Comment by Gerrit Updater [ 29/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/35961
Subject: ~~LU-8465~~ e2fsck: merge bitmaps after thread completes
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 06c6eca0e1447669e20a4cd9c10421610a4b1a9d

Comment by Andreas Dilger [ 29/Aug/19 ]

I used 128 threads to do the pass1 check. And only 13 threads have real work to do. It took at most 800 seconds for the slowest thread to finish.

While there is a limit to the parallelism on a given system, if you split the workload into flexbg_count chunks (Lustre OSTs use flexbg_count=256 so that inode tables and block bitmaps are allocated in 1MB chunks on disk) then the workload could be spread over 100 threads in your test case, and the workload would likely be much more even. Each thread would process groups numbered:

group = n * num_threads + flexbg_count * thread + [0..flexbg_count - 1], n: 0, 1, 2, ...

In theory, even with the 13 threads doing an even amount of work (I'm assuming a "base" of ~215s for each thread, and the 5000s above that is "work") then pass1 would finish in about 635s. If 104 threads could participate evenly (at 256 groups/thread no more will have work to do), then pass1 could finish in about 265s.

Comment by Gerrit Updater [ 30/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36001
Subject: ~~LU-8465~~ e2fsck: do not change global variables
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: a8ef87bfc39ee35441f2763de44c154ddb818716

Comment by Andreas Dilger [ 30/Aug/19 ]

Li Xi, have you looked at using pthreads for the process control and communication between threads rather than the current fork+exec?

Looking at the e2fsck performance results, it seems that pass3 takes as long to complete as pass1, so it seems that once we have pass1 running in parallel then we also need to improve pass3 in order to significantly reduce the total runtime.

With pass3 (pathname connectivity), it will definitely need more fine-grained communication between the threads to exchange results, and also also shared memory to avoid significant duplication of effort or memory usage.

Comment by Li Xi [ 30/Aug/19 ]

Li Xi, have you looked at using pthreads for the process control and communication between threads rather than the current fork+exec?

Currently, my patches are already using pthreads, not fork+exec.

Looking at the e2fsck performance results, it seems that pass3 takes as long to complete as pass1, so it seems that once we have pass1 running in parallel then we also need to improve pass3 in order to significantly reduce the total runtime.

I don't think so. Pass3 only takes a 0.02 seconds according to https://jira.whamcloud.com/browse/LU-8465?focusedCommentId=253670&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-253670

The logs of e2fsck is not super clear though:

Pass 3: Checking directory connectivity
Peak memory: Memory used: 75116k/337408k (27172k/47945k), time: 3879.43/1568.92/35.05
^^^^^^^^^^^^^^^^ This is not the time/memory usage for Pass3, this is the time since fsck starts.
Pass 3: Memory used: 75116k/337408k (25930k/49187k), time: 0.02/ 0.03/ 0.00
^^^^^^^^^^^^^^^^ This is is the time/memory usage for Pass3
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s

Comment by Gerrit Updater [ 30/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36004
Subject: ~~LU-8465~~ e2fsck: optimize the inserting of dir_info_db
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 9022f19466a871305b77b7605dc968545b20f6a7

Comment by Li Xi [ 30/Aug/19 ]

Patch https://review.whamcloud.com/36004 adds e2fsck_dir_info_min_larger_equal(). test_max_less_equal.c file tests that min_larger_equal() and max_less_equal() is correct.

Comment by Gerrit Updater [ 30/Aug/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36006
Subject: ~~LU-8465~~ e2fsck: debug e2fsck_insert_dir_info
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: b4d83091ca90ad1a8acb58cb03a2ffdd093b612c

Comment by Gerrit Updater [ 02/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36018
Subject: ~~LU-8465~~ e2fsck: merge dir_info after thread finishes
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: f67bccf841a002ff06f3fc2d5a9df54dd1a21c2e

Comment by Gerrit Updater [ 02/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36019
Subject: ~~LU-8465~~ e2fsck: free bitmaps even thread quit
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: c6b9ad0533ec66b75c62390eccd5043f5773267a

Comment by Gerrit Updater [ 02/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36020
Subject: ~~LU-8465~~ e2fsck: rbtree bitmap for dir
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4efa96732f0c9ba625caee91a1e9e8c320b770c1

Comment by Gerrit Updater [ 03/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36026
Subject: ~~LU-8465~~ e2fsck: merge badblocks after thread finishes
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: c5214a132138451f90f5d5ce966df02ce534ed7d

Comment by Gerrit Updater [ 04/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36044
Subject: ~~LU-8465~~ e2fsck: merge inode_link_info after thread finishes
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 46db7ec72a7e5fd5160cf65e798f0a9f1b3a44ed

Comment by Gerrit Updater [ 05/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36053
Subject: ~~LU-8465~~ e2fsck: merge dblist after thread finishes
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 37b1bc16ea047951910c39574ff757d9c4431379

Comment by Gerrit Updater [ 05/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36054
Subject: ~~LU-8465~~ e2fsck: add debug codes for multiple threds
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: a7a62d8286c833a6b04cc3d356465fe8a448067c

Comment by Gerrit Updater [ 06/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36079
Subject: ~~LU-8465~~ e2fsck: do not start thread without any work
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 44ea3c360935f4bd5c19bc5842181cedb7662769

Comment by Gerrit Updater [ 06/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36080
Subject: ~~LU-8465~~ e2fsck: fixed a bug of rb_merge_bmap
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 30f7489acaceb27f4e36a1414712962d3223e7d5

Comment by Gerrit Updater [ 06/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36081
Subject: ~~LU-8465~~ e2fsck: only reserve block for root in the first thread
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: dcf826973e7e07369ac5e8b9f942bfe099de4965

Comment by Gerrit Updater [ 06/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36083
Subject: ~~LU-8465~~ e2fsck: fix problem of pthread debug
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 984d49cc207e2f2d617ce4ca58daf47e15291cbb

Comment by Gerrit Updater [ 06/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36084
Subject: ~~LU-8465~~ e2fsck: clear the reserved blocks when restart
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: b89f249ba2e150a4dc2906ba48a486e21fd9e6ad

Comment by Gerrit Updater [ 08/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36097
Subject: ~~LU-8465~~ e2fsck: merge counts when threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4f1c0aa71ff514a0d131c3facb59eb54228a7188

Comment by Gerrit Updater [ 08/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36098
Subject: ~~LU-8465~~ e2fsck: merge fs flags when threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: b7334aea79e73ec11e17bf06b55add9f36d9096d

Comment by Gerrit Updater [ 08/Sep/19 ]

Li Xi (lixi@ddn.com) uploaded a new patch: https://review.whamcloud.com/36099
Subject: ~~LU-8465~~ e2fsck: merge icounts after thread finishes
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 1f15801a512ac3d3bf671f5ec162d14b964b85f1

Comment by Wang Shilong (Inactive) [ 19/Nov/19 ]

Looked at current patches, i think it might have a design problem that was ignored, e2fsck works as following:

e2fsck_pass1->e2fsck_pass1e->e2fsck_pass2->e2fsck_pass3->e2fsck_pass4->e2fsck_pass5

And existed patches flow was trying to use pthreads in e2fsck_pass1(), then merge results after e2fsck_pass1 and continue.
The problem is there isn't any memory sharing module to make sure how this could be worked?

As in the threads of e2fsck_pass1() we will malloc memory which will be used later, the current model as i understand is:

Parent process->pthread_create sub thread->sub thread malloc memory->Parent process try to access sub thread memory?

I don't think this will work? We might need some good memory sharing frame to work this out?

Comment by Li Xi [ 22/Nov/19 ]

> The problem is there isn't any memory sharing module to make sure how this could be worked?

Shilong, there will be certainly some need to sync between different threads in some circumstances. For example, bitmap is updated by a thread, that means the bitmap would need to synced to all of the other threads too. That part is not finished by the patches yet.

In order to sync between threads, we might need to change the logic of the e2fsck_pass1() a little bit to have some sync step before trying to fix the problem. There are other ways, I am not sure which one is better though.

> Parent process->pthread_create sub thread->sub thread malloc memory->Parent process try to access sub thread memory

No, all the memory allocated by the sub threads will be merged to the global thread, and then freed.

Comment by Gerrit Updater [ 26/Nov/19 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36862
Subject: ~~LU-8465~~ e2fsck: fix to return ENOMEM in alloc_size_dir()
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 7ece6455f4794a967cbc6313de7760707d23fbf0

Comment by Gerrit Updater [ 26/Nov/19 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36863
Subject: ~~LU-8465~~ e2fsck: fix use after free in calculate_tree()
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: c98eea9615b7212d816087b62778ff1588d314ea

Comment by Andreas Dilger [ 12/Dec/19 ]

I was reading the pthread(7) man page, and it states:

POSIX.1 specifies a set of interfaces (functions, header files) for
threaded programming commonly known as POSIX threads, or Pthreads. A
single process can contain multiple threads, all of which are executing
the same program. These threads share the same global memory (data and
heap segments), but each thread has its own stack (automatic variables).

So we should be able to have e.g. a shared block and inode bitmaps as long as they are allocated before the threads are forked (so they all know where the bitmaps are allocated), and we use locking to avoid multiple threads updating the same structures at the same time (words or rbtree).

We can use a scalable pthread_mutex() implementation for updating the block bitmaps (e.g. block group number hashed across 4*num_threads locks, similar to struct blockgroup_lock *s_blockgroup_lock in the kernel). The inode bitmap should be relatively uncontended, since we will be splitting whole block groups between threads, but it should still use a scalable pthread_mutex_lock() for access like with the block bitmap. I think it makes sense to lock whole groups rather than individual words, since it is likely extents will span hundreds of blocks, and threads will be processing thousands of inodes in the same bitmap.

I think there are enough abstraction hooks in libext2fs for the bitmap handling that we can register our own callbacks from the code to avoid embedding pthreads into the core library (which will otherwise cause problems for non-pthread users). It may be a bit tricky to consolidate the rbtree bitmap with parallel access, but we could potentially lock the nodes of the rbtree similar to htree lock instead of the groups?

Comment by Andreas Dilger [ 12/Dec/19 ]

Note, in addition to the bitmaps, it probably makes sense to push the problem handling code to the main thread. This could be done with a producer-consumer model. While single-threading the error handling is not as fast as doing it in parallel, I think it may too complex to decide which problems can be fixed by a thread (and in parallel). It may be that all pass1 errors can be handled immediately (i.e. they only affect the current inode and mark its blocks set in the bitmap), but this would need to be verified.

Comment by Gerrit Updater [ 16/Dec/19 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37022
Subject: ~~LU-8465~~ e2fsck: Don't use EXT2_FLAG_EXCLUSIVE for multiple threads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: efc91105a42c60fad824db4a6b0759247628b3b0

Comment by Gerrit Updater [ 16/Dec/19 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37024
Subject: ~~LU-8465~~ e2fsck: merge dx_dir_info
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 244851fbb6d644a00977a29e4a7a4cd6548b499a

Comment by Gerrit Updater [ 16/Dec/19 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37023
Subject: ~~LU-8465~~ e2fsck: fix wrong memory access in ext2fs_merge_dblist()
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 95939129d422dd6cb1547a6aa8fc0ff69104a82a

Comment by Gerrit Updater [ 16/Dec/19 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37026
Subject: ~~LU-8465~~ e2fsck: split and merge invalid bitmaps
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 76b103f3c503554813374a040165776c074098e1

Comment by Gerrit Updater [ 16/Dec/19 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37025
Subject: ~~LU-8465~~ e2fsck: make threads splitting aware of flex_bg
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4a129c8c4e9616cb1e29da503f524405c19e5ff1

Comment by Andreas Dilger [ 18/Dec/19 ]

Since (AFAIK) the block and inode bitmap code is using an rbtree, I think there are two options for how to handle these bitmaps between threads:

a single global bitmap that has locking to protect all thread access (the locking itself could be hashed to avoid the majority of contention, but potentially every bit modified would need to get a lock). The benefit of this model is that it is easier to switch between threaded and serialized operation.
each thread has a separate bitmap and there is a function to merge bitmaps at the end of pass1. The drawback of this approach is that the bitmap is not visible to other threads until they have finished their work, which may prevent pass1 repair actions, but maybe they do not depend on these bitmaps to do anything? Possibly inode badness depends on a global badness value, but this is unlikely to be a performance bottleneck since it is rarely updated.

It would probably be worthwhile to run a quick benchmark to call pthread_mutex() in a loop on an uncontended lock to see how many locks/sec it can get,then run the same for two threads on the same lock to see what the aggregate locks/dev they can get, and two threads locking different locks. This will help make the decision whether the locking is expensive and should be avoided at all costs, or whether lock hashing would be enough to avoid contention in most uses.

Comment by Gerrit Updater [ 21/Feb/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37657
Subject: ~~LU-8465~~ e2fsck: merge group desc
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4df9839097114c3b91429f7bae8fe5767a457270

Comment by Gerrit Updater [ 21/Feb/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37658
Subject: ~~LU-8465~~ e2fsck: only allow first thread to recreate resize inode
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: d9ec4dc8f8644cba37e6846b6ca1ab49d9344a64

Comment by Gerrit Updater [ 03/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37782
Subject: ~~LU-8465~~ e2fsck: merge dirs_to_hash when threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: dd2f378a63c9bbadc7b462ddf53f14489068b817

Comment by Gerrit Updater [ 03/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37784
Subject: ~~LU-8465~~ e2fsck: fix to avoid splitting too much for small fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: da3471ebb572ee54ae4ea0a072c00df4ec70682b

Comment by Gerrit Updater [ 03/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37783
Subject: ~~LU-8465~~ e2fsck: merge context flags properly
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: c6a049258fa23fa153073a8e04bad0f77f8dec42

Comment by Gerrit Updater [ 03/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37785
Subject: ~~LU-8465~~ e2fsck: fix to use correct list in merge_u32_list()
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: edda4004d9885c760eb0a51328c8c508e7c5cfcd

Comment by Wang Shilong (Inactive) [ 03/Mar/20 ]

Currently, i've tested the whole series patches with number of thread 8, and all test passed with e2fsprogs, but there are still some problems that i want to discuss here:

actually, for pass1, parallel repair is possible, as mostly of repiar for it is just inode rewrite(flags, size etc), but there are still some possible issues to be global:

1) found_block, dup_blocks.
This is used to fix duplicated blocks, this need be global between different threads as scanning going on, so that it could find duplicated blocks correctly, otherwise, after merging finish, it just mean no duplicated blocks within every thread context, we still need find every blocks to check which might make things complex.

2) block allocations/free,
This doesn't happen much, but still possible for pass1, etc recreate resize inode, relocate bitmaps etc, i think we shall make this operation exclusive for each threads.

3)superblock
This is global information, it is nature to make it global rather than merge it after threads finish, especially, there are a lot of areas inside it.

So potentially, there could be two ways to address issues like this:
1) We only push all fixes to master thread(let's say it as thread 0), all other threads just try to do reaonly open and try readonly check once it find some errors, push it to master thread and let master thread do real fix.

Good benefits of this is:
1.1 we could keep EXCLUSIVE open flag for e2fsck.
1.2 it looks safer to guarantee correctness, since only one writer.

Bad side is:
1.3 more modifications to core codes of pass1, and need some proper memory sharing model to guarantee correctness, and there might be some gap from finding problem and fixing it, might break some depedency for this though.

2) Allow parallel fix of pass1(what current series did)

Good benefits of this:
2.1) Less changes to core codes which might make it less error-prone.
2.2) real parallel fix which is faster.

Bad sides.
2.3) Break EXCLUSIVE open of e2fsck if we try multiple threads.
2.4) need different mutex/read/write locks to protect global variables, and serialize some criticial fixes between different threads.

Any ideas or input what is better idea that we could go?

Comment by Li Xi [ 03/Mar/20 ]

Comparing to parallel checking, parallel fixing isn't so important, because 1) most file systems are not broken and 2) most part of a broken file system are not broken 3) it is more acceptable to fixing a broken file system more slowly but less risky than more quickly but more risky.

So, if parallel fixing is not super important, I think the top challenge here is how to write trustable fix codes. So, if putting all fixing to a single thread is simpler and clearer, then we should do that. But I agree with you that this is not going to be the case. Each thread has its own checking and fixing process. Fixing can only happen naturely when checking finds a problem. I am not sure it is easy to put all codes into a single thread.

And fixing the problems in parallel by multiple threads look dangerous too. The reason is, essentailly multiple threads write to the same shared device. And if locks are not protecting the ciritical region correctly, data will be corrupted.

So, instead of either the upper two ways, I think maybe the following ways is better. Each thread do the check and fix by itself. But whenever the thread need to change/write anything (either in memory or to disk) that could be shared by other threads, it should do that exclusively in a special fixing context. Before the changing/fixing, all threads might need to sync with the fixing thread on the shared information (block bitmap etc). And after the changing/fixing, all threads might need to sync the fixing thread again so everyone agree on the change. And during the changing/fixing, no other thread do anything including doing I/O, so that the fixing thread can do I/O exclusively or change anything it wants.

I think we can start by strict policy first, i.e. anything that we are not sure whether is shared by others, we can use the fixing context to protect it. When we know more about the logic details and performance bottlenecks, we can future use other ways to remove possible bottlenecks. And in the end, the parallel fixing process will be turstable, stable and at the same time quick.

Comparing to fixing the problem in a dedicated thread, we don't need to change the current fixing codes a lot. We just need to add codes to enter and quit the context.

Comparing to adding a lot of mutex/read/write locks, this doesn't require deep understanding of all details. And we are pretty sure the fixing and modification won't cause any data corruption due to race or overlapping.

What do you think?

Comment by Wang Shilong (Inactive) [ 04/Mar/20 ]

All threads might need agreed changes before going on, this might be problem, actually this is exclusive operation:

Considering superblock cases, we have 100 threads, eveytime, we need dirty superblock, we need copy 99 * 4096 buffer head, maybe just one mutex to hold on whenever fix is needed for possible shared contents modified is simper, in this case, thread contexts
need share memory from global context for some cases.

Comment by Andreas Dilger [ 04/Mar/20 ]

I agree with Li Xi, that the speed of fixing problems is much less critical than the scanning of correct parts of the filesystem, and the correct fixing of errors. The only main slowdown in pass1 is the duplicate blocks handling case (pass1b/c/d), and I think we should generally try to avoid that (adding/improving the "inode badness" patch to better catch cases where one inode is a duplicate of another).

The vault20-pFSCK-domingo.pdf code took the approach where the "fix" was done in the context of each thread, but other threads were blocked during the fixing process. This serialized the threads and prevented parallel fixing of individual problems, but each thread could be scanning in parallel. That said, it wasn't clear what the memory model is in their pFSCK (globally shared or per-thread). That might imply that a mutex covering the "Fix? y" case would be enough to block out other threads from repairing the filesystem concurrently, but it wouldn't prevent them from changing other parts of the filesystem (e.g. bitmaps, arrays) that might be needed by the thread doing the repair.

Comment by Gerrit Updater [ 06/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37825
Subject: ~~LU-8465~~ e2fsck: split and merge quota context
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: edbed5c9bce76994735f8e1a5360b9986db8b4c3

Comment by Gerrit Updater [ 06/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37826
Subject: ~~LU-8465~~ e2fsck: serialize fix operations
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: e35249ed353ef7d9f1b712de8847415e4a6376c9

Comment by Gerrit Updater [ 10/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37854
Subject: ~~LU-8465~~ e2fsck: share one global superblock
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4051baea11a3eca68a860b3898d19d53c37bc5fd

Comment by Gerrit Updater [ 10/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37855
Subject: ~~LU-8465~~ e2fsck: share global group desc
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 8834c3d449530f4e3f40b6ef5fe81db4576105d2

Comment by Gerrit Updater [ 10/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37856
Subject: ~~LU-8465~~ e2fsck: move some fixes out of parallel pthreads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 1464d92d2a346942c5da567348cb015513e2891e

Comment by Gerrit Updater [ 10/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37857
Subject: ~~LU-8465~~ e2fsck: fix to protect EA checking
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 617a0a257c171314e87b76ddc436694954a7b70a

Comment by Gerrit Updater [ 10/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37858
Subject: ~~LU-8465~~ e2fsck: fix to merge quota updates
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 1bc8b606300ee2ec669c5a20a686825c19001667

Comment by Gerrit Updater [ 10/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37859
Subject: ~~LU-8465~~ e2fsck: allow admin specify number of threads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 41326f0d7d10e5aab05f8dd45c9c781d7cd46209

Comment by Gerrit Updater [ 11/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37885
Subject: ~~LU-8465~~ e2fsck: fix to calculate invalid_bitmaps
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: eeca027bed3bb2fb4f297303b02cf9b08fdda772

Comment by Gerrit Updater [ 11/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37887
Subject: ~~LU-8465~~ e2fsck: fix readahead for pfsck of pass1
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: bfbebd4bc5a811ca3b9c132de64a661c441de5c7

Comment by Gerrit Updater [ 11/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37886
Subject: ~~LU-8465~~ e2fsck: fix to get_lost_and_found before threads start
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 77ead2e939c2de1c1051fdba313259a1ac18592e

Comment by Gerrit Updater [ 11/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37884
Subject: ~~LU-8465~~ e2fsck: kickoff mutex lock for block found map
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 7f0b446dbbac18768c677b518282212ed4073cf3

Comment by Gerrit Updater [ 12/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37905
Subject: ~~LU-8465~~ e2fsck: kick off ea mutex lock from pfsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 86ef85e6f2e8a5f39209227c92efbf675fd1ac70

Comment by Gerrit Updater [ 20/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37995
Subject: ~~LU-8465~~ e2fsck: merge encrypted dir lists after thread finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 451649cf5292859ee4f3061798a801946b2adec5

Comment by Gerrit Updater [ 20/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37999
Subject: ~~LU-8465~~ e2fsck: reset lost_and_found after threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: b527050b055605273c7a2a467df98cfa3bf7c1ac

Comment by Gerrit Updater [ 20/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37997
Subject: ~~LU-8465~~ e2fsck: merge min_extra_isize after threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 57f077a370db8c9b20566cf8db8af4c187f59593

Comment by Gerrit Updater [ 20/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37996
Subject: ~~LU-8465~~ e2fsck: simplify e2fsck context merging codes
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 8b193d2dd65d767699414d33d7ff252c72814913

Comment by Gerrit Updater [ 20/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/37998
Subject: ~~LU-8465~~ e2fsck: merge options after threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 52f6d7e5428bf25ba1390246aba0e502ed590edd

Comment by Gerrit Updater [ 20/Mar/20 ]

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/38000
Subject: ~~LU-8465~~ e2fsck: merge extent depth count after threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 6b81e80b90bf20450348b79f5e07ee53d9f60e20

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39840
Subject: ~~LU-8465~~ e2fsck: remove unused fs_ext_attr_inodes/blocks
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 1a8d5b742965e45d428fa605ad3ea8e35f009f37

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39841
Subject: ~~LU-8465~~ e2fsck: merge icounts after thread finishes
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: d2660c3a05b331fda3ea143dda50b9a9adb0dde5

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39843
Subject: ~~LU-8465~~ e2fsck: merge inode_badness after threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 7ba7c8aa34182777e5482b5d1e95daa995869e17

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39845
Subject: ~~LU-8465~~ e2fsck: only setup threads if -m option required
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 9bfbd37b98fc5b0bb65c9962135ca6a50c96a5d6

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39844
Subject: ~~LU-8465~~ e2fsck: set E2F_FLAG_ALLOC_OK after threads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 9ef377d98d85a717485fe7189472d0777832d9b5

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39842
Subject: ~~LU-8465~~ e2fsck: merge inode_bad_map after threads finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: f82ee5ab3e7eee9b289245dac1fcdf28ab591a3d

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39846
Subject: ~~LU-8465~~ e2fsck: wait fix thread finish before checking
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: c8ecbc8f5ebb83d891366a4d027aea7216c2c13a

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39847
Subject: ~~LU-8465~~ e2fsck: fix merge dblist and badblocks properly
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: fdee28ee36c45284a47e174570e11ed7c5976520

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39848
Subject: ~~LU-8465~~ e2fsck: fix to free icache leak
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 8d83b8c210b118a48b8fab34f2def625b07776d8

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39850
Subject: ~~LU-8465~~ ext2fs: fix to avoid invalid memory access
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 7831f11762c38d358644022c67bfffef1e794264

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39849
Subject: ~~LU-8465~~ e2fsck: avoid too much memory allocation for pfsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 0ade56d171a0c5a4b487d8225dff6a4992375fb8

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39851
Subject: ~~LU-8465~~ ext2fs: parallel bitmap loading
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: c91e78e9ea9bf6248c1f3434e1c31ed5987888f0

Comment by Gerrit Updater [ 09/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39852
Subject: ~~LU-8465~~ e2fsck: adjust number of threads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: e957e7c1903a5b80af133d8fbe9cce573abe48b3

Comment by Gerrit Updater [ 10/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39874
Subject: ~~LU-8465~~ e2fsck: skip mmp update for multiple threads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 5e6e07ce8c7efaf0f4223260e8e49ecc4f4f057f

Comment by Gerrit Updater [ 15/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39914
Subject: ~~LU-8465~~ e2fsck: make default smallest RA size to 1M
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: d1df7948228334274d0dd5f86d0fc5e2aae8ab8d

Comment by Gerrit Updater [ 18/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35684/
Subject: ~~LU-8465~~ e2fsck: cleanup struct e2fsck_struct
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 8ba5db339a5fbf8e2dbde1b517879c0a7de6f5bb

Comment by Gerrit Updater [ 18/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/39840/
Subject: ~~LU-8465~~ e2fsck: remove unused fs_ext_attr_inodes/blocks
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 828f7975917c5351ba2616ae28aa737857b605a0

Comment by Gerrit Updater [ 18/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35689/
Subject: ~~LU-8465~~ e2fsck: add -m option for multithread
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: e80445181c12f55a61a6e4d4b178d2960d0e403d

Comment by Gerrit Updater [ 18/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35690/
Subject: ~~LU-8465~~ e2fsck: copy context when using multi-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 59c8379f2b51885ee74b248ab93bdf742c1eb96d

Comment by Gerrit Updater [ 18/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35696/
Subject: ~~LU-8465~~ e2fsck: copy fs when using multi-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 7432f421b844b5cc6c84db05454d7cfdba1d449f

Comment by Gerrit Updater [ 18/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35698/
Subject: ~~LU-8465~~ e2fsck: copy dblist when using multi-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: e5058d7508eec4997fb0a0858d4ad80ee9206798

Comment by Gerrit Updater [ 18/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35701/
Subject: ~~LU-8465~~ e2fsck: clear icache when using multi-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: d8c6866b683bf0007ef59306ff667cd3de57fbe8

Comment by Gerrit Updater [ 18/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35710/
Subject: ~~LU-8465~~ e2fsck: add assert when copying context
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: cbf3c4205eb174b2141d2b160557823b89632d40

Comment by Gerrit Updater [ 23/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/40014
Subject: ~~LU-8465~~ e2fsck: clear icache when using multi-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 542a0eeb8f44d9f0a0ceef7a40f3a052a527f86c

Comment by Gerrit Updater [ 23/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/40015
Subject: ~~LU-8465~~ e2fsck: add assert when copying context
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: e1acbb00bb1c8635ed0626dbf771b28b8449ce36

Comment by Gerrit Updater [ 23/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/40017
Subject: ~~LU-8465~~ test: add pfsck test
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 8f12d37566dec681ebda800a9f5ea417190b6db9

Comment by Gerrit Updater [ 24/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/40021
Subject: ~~LU-8465~~ configure: enable pfsck by default
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 592c95b285837cf30b0c16780a6fbe6aa9ebac5a

Comment by Gerrit Updater [ 24/Sep/20 ]

Wang Shilong (wshilong@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/40024
Subject: ~~LU-8465~~ e2fsck: reset @inodes_to_rebuild if restart
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 276cff391901e38601728cab11f7638d3732b4b2

Comment by Gerrit Updater [ 25/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/40014/
Subject: ~~LU-8465~~ e2fsck: clear icache when using multi-thread fsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 321ed5ab1d11ca2e74adbd84222f96a6dc180da9

Comment by Gerrit Updater [ 25/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/40015/
Subject: ~~LU-8465~~ e2fsck: add assert when copying context
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 18a7cf47f057c152ba76f5bf82b0ff4504ab116a

Comment by Gerrit Updater [ 25/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35712/
Subject: ~~LU-8465~~ e2fsck: copy bitmaps when copying context
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 87444962ca6ef43f3f2fca6a5f8c3750961f1237

Comment by Gerrit Updater [ 25/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35726/
Subject: ~~LU-8465~~ e2fsck: copy badblocks when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 1d9178988c187efa610c97c36ee0d6c3d410f947

Comment by Gerrit Updater [ 25/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35763/
Subject: ~~LU-8465~~ e2fsck: open io-channel when copying fs
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 0db312fbbc5d6a7b6cd5fa43b513e9fdfd9a717c

Comment by Gerrit Updater [ 25/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35780/
Subject: ~~LU-8465~~ e2fsck: create logs for mult-threads
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 2023afea8aa825da45de7a66cee9e8182dd79cf9

Comment by Gerrit Updater [ 25/Sep/20 ]

Andreas Dilger (adilger@whamcloud.com) merged in patch https://review.whamcloud.com/35793/
Subject: ~~LU-8465~~ e2fsck: optionally configure one pfsck thread
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 727154449dd68c7b3197d8cc4d0558f3826dc692