[LU-12103] Improve block allocation for large partitions

• Type: Improvement
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.4
• Assignee: Artem Blagodarenko (Inactive)
• Reporter: Artem Blagodarenko (Inactive)

Description

Block allocator uses some heuristic while chooseing group to allocate new blocks. This works good in most cases, but take a time for large low-free-space partition. The algorithm should be adjusted for this special case.

Issue Links

• is related to: LU-12335 mb_prealloc_table table read/write code is racy
• is related to: LU-12970 improve mballoc for huge filesystems
• is related to: LU-12988 improve mount time on huge ldiskfs filesystem
• is related to: LU-14305 add persistent tuning for mb_c3_threshold
• is related to: LU-8365 Fix mballoc stream allocator to better use free space at start of drive
• is related to: LU-12345 backport - ext4: optimize ext4_find_delalloc_range() in nodelalloc mode
• is related to: LU-12801 Port "ldiskfs: don't search large block range if disk full" to RHEL 8

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36681/
Subject: LU-12103 ldiskfs: don't search large block range if disk full
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 810a952303969ca0ee01639a5408ff2f0e3456d9

[Andreas Dilger]

Hi Artem, I noticed that this patch was only added to the rhel7.6 series, but not the rhel7.7 and rhel8.0 series. Could you please submit a patch to add ext4-simple-blockalloc.patch to these newer series.

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36681
Subject: LU-12103 ldiskfs: don't search large block range if disk full
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 0da3a44425f5ab1c0417663281f7cc626f99b675

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35180/
Subject: LU-12103 ldiskfs: don't search large block range if disk full
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 95f8ae5677491508ae7182b4f61ead3d413434ae

[Artem Blagodarenko (Inactive)]

green, I added the issue is about porting to RHEL8 - LU-12801

[Gerrit Updater]

Artem Blagodarenko ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35180
Subject: LU-12103 ldiskfs: don't search large block range if disk full
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5444c2b2d17a58f7b0d2d8aeb23b652ae8d6ecd4

[Artem Blagodarenko (Inactive)]

Here is summary from EXT4 developers call descussion last Thursday.

• EXT4 users prefer automatic block allocator settings adjusting 
• Current versions of skipping loops patch is not interesting, because requires manual setting.
• Some heuristic that change allocator behaviour is preferred.

My next steps:

• Continue discussion in EXT4 email list to find the best heuristic
• Upload current patch version to the Gerrit

[Andreas Dilger]

I agree that it makes sense to get the patches reviewed and accepted upstream if possible, but after that it might take several years before the change is available in a vendor kernel, so it would make sense to have an ldiskfs patch as well.

Also, in some cases, patches that improve performance and/or functionality still do not get accepted upstream because of various reasons, so in this case it would still make sense to carry this patch in the Lustre tree because it mostly affects very large OST filesystems.

[Artem Blagodarenko (Inactive)]

Hello adilger,

>I didn't see a copy of the patch in Gerrit. Have you submitted it yet?

Do we need LDISKFS patches here? The right way is land it to ext4 directly. I am going send new patches series, with test results and debugfs fake fragmentation patch for tests.