[LU-12345] backport - ext4: optimize ext4_find_delalloc_range() in nodelalloc mode

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.5
• Labels: patch
• Severity: 3
• Assignee: Artem Blagodarenko (Inactive)
• Reporter: Artem Blagodarenko (Inactive)

Description

From 8c48f7e88e293b9dd422bd8884842aea85d30b22
..
We found performance regression when using bigalloc with "nodelalloc"
(1MB cluster size):
1. mke2fs -C 1048576 -O ^has_journal,bigalloc /dev/sda
2. mount -o nodelalloc /dev/sda /test/
3. time dd if=/dev/zero of=/test/io bs=1048576 count=1024
The "dd" will cost about 2 seconds to finish, but if we mke2fs without
"bigalloc", "dd" will only cost less than 1 second.
 
 

Issue Links

• is related to: LU-12103 Improve block allocation for large partitions
• is related to: LU-16286 nodelalloc optimization is missing for RHEL8 kernels causing performance drop

Comments

[Gerrit Updater]

{}Chris Horn <[e-mail]> uploaded a new patch:{-} [-https://review.whamcloud.com/c/fs/lustre-release/+/58806-]
Subject: LU-12345 lnet: Fix lnetctl fault commands
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cb3eeba1c9065c2bd67b734204884953e71f747a

Sorry, pushed patch against wrong ticket.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37538/
Subject: LU-12345 ldiskfs: optimize nodelalloc mode
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: f91552d85cd086b37f78524113cadf799c045220

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37538
Subject: LU-12345 ldiskfs: optimize nodelalloc mode
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: bbf850c5ac55b631081848a67831d23ac4a241a8

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34982/
Subject: LU-12345 ldiskfs: optimize nodelalloc mode
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: af48ae8bff289b2bc083a888efeafa3c48df91e2

[Artem Blagodarenko (Inactive)]

adilger going to make bigalloc testing in near future.

[Andreas Dilger]

Other than this (obviously significant) performance problems, have you seen other problems with bigalloc? I'm worried that there are hidden problems because the osd-ldiskfs code is dealing with blocks instead of chunks in various places (chunk, C2B, and B2C do not appear anywhere in that code).

[Gerrit Updater]

Artem Blagodarenko ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34982
Subject: LU-12345 ldiskfs: optimize nodelalloc mode
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7e74f2751f8cfdce2915d9b92259cdfc78d3a8d4

[Artem Blagodarenko (Inactive)]

adilger,

There is "ext4: optimize ext4_find_delalloc_range() in nodelalloc
    mode" patch that prevents executin
    ldiskfs_es_find_delayed_extent_range() if no delalloc enabled. The
    same optimization also added for ldiskfs_find_delayed_extent()
    function that improve performance dromaticaly.
 
    Here is results of testing on two node system.
    Without the patch:
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.00    0.00   56.30    0.06    0.00   43.63
 
     Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
     avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
     sds               0.00     0.00    0.00 1174.00     0.00     4.59
     8.00     0.84    0.71    0.00    0.71   0.01   1.20
 
     With patch:
     08/29/2018 01:13:22 AM
     avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                0.00    0.00    4.13   30.37    0.00   65.50
 
    Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s      wMB/s
    avgrq-sz avgqu-sz   await r_await w_await  svctm %util
    sds               0.00     0.00    0.00 54117.82     0.00     211.43
    8.00   152.59    2.82    0.00    2.82   0.02 99.01

[Artem Blagodarenko (Inactive)]

> I don't think osd-ldiskfs is using delalloc? This may affect local ext4 performance but I don't think it will affect Lustre.

During IOR single thread 4K random re-write over 16GB file on one of our machine 
    the CPU usage on the OSS is 100% and resulting IOPS are extremely
    low. Here is a snippet of perf report:
 
   |-89.87%- ldiskfs_es_find_delayed_extent_range
    |          ldiskfs_fiemap
    |          osd_is_mapped
    |          osd_declare_write_commit
    |          ofd_commitrw_write
    |          ofd_commitrw
    |          tgt_brw_write
    |          tgt_request_handle
    |          ptlrpc_server_handle_request
    |          ptlrpc_main
    |          kthread
    |          ret_from_fork