[LU-8465] parallel e2fsck performance at scale

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0
• Labels: ldiskfs scalability support
• Assignee: Wang Shilong (Inactive)
• Reporter: Artem Blagodarenko (Inactive)

Description

Mentioned in LU-7592 by Andreas Dilger

e2fsck performance will become an issue at this scale, and it would likely need to be parallelized to be able to complete in a reasonable time. It could reasonably expect multiple disks at this scale, so having larger numbers of IOs in flight would help, as would an event-driven model with aio that generates lists of blocks to check (itable blocks first), submits them to disk, and then processes them as they are read, generating more blocks to read (more itable blocks, indirect/index/xattr/directory blocks, etc), repeat.

Issue Links

• is related to: LU-5949 Verify e2fsck inode badness is working for duplicate blocks
• is related to: LU-14126 parallel e2fsck does not work well with MMP
• is related to: LU-14894 Parallel pass2 support for e2fsck
• is related to: LU-14213 enable parallel e2fsck by default
• is related to: LU-14679 parallel e2fsck for pass2 directory scanning
• is related to: LU-11882 OST recreated objects gets badness mark from e2fsck
• is related to: LU-12051 ldiskfs directory shrink
• is related to: LU-7370 improve e2fsck lost+found insert performance
• is related to: LU-7592 Сhange force_over_128tb lustre mount option to force_over_256b for ldiskfs
• is related to: LU-8974 Сhange force_over_256tb lustre mount option to force_over_512tb for ldiskfs

Comments

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44011
Subject: LU-8465 e2fsck: merge expand_eisize_map properly
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 3055d6fa13f17f9794e3e8b7687ab69d59702bd8

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44010
Subject: LU-8465 e2fsck: merge casefolded dir lists after thread finish
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: 4aff8d51e30b20c4da49434a50ca49a104c0473e

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/43129/
Subject: LU-8465 ext2fs: RO open for bitmap loading
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: c0f60f896182d26711c7d8d064dc95471b0d9829

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/40070/
Subject: LU-8465 tests: enable pfsck if possible
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 55a4357424c5206910928f27f66db62465581150

[Andreas Dilger]

The e2fsprogs-1.45.6.wc2 build is available at https://downloads.whamcloud.com/public/e2fsprogs/1.45.6.wc2/

[Wang Shilong (Inactive)]

I think it fine to close this ticket, and we could open new ticket for further work.

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/40081/
Subject: LU-8465 build: update version to 1.45.6-wc2
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 0b7bcc70a54425f53407fe7825caa4671b5bd082

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/40081
Subject: LU-8465 build: update version to 1.45.6-wc2
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: f2ddfd30f03c71e6ff7423ff816160e97a073baa

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/40075/
Subject: LU-8465 e2fsck: misc cleanups for pfsck
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: d9400e590b9f1ca026241501b22c7674d774da22

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/40071/
Subject: LU-8465 ext2fs: fix to set tail flags with pfsck enabled
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set:
Commit: 6505169f9a2a2cce0a8c383c1aa10292e01c6f4a