[LU-14641] per extents bytes allocation stats

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.15.0
• Assignee: Wang Shilong (Inactive)
• Reporter: Wang Shilong (Inactive)

Description

Seeing debug messages on the console during large RPC writes:

Apr  3 01:29:08 foss01 kernel: Lustre: 12536:0:(osd_handler.c:1947:osd_trans_start()) work-OST0004: credits 24347 > trans_max 12800
Apr  3 01:29:08 foss01 kernel: Lustre: 12536:0:(osd_handler.c:1876:osd_trans_dump_creds())   create: 0/0/0, destroy: 0/0/0
Apr  3 01:29:08 foss01 kernel: Lustre: 12536:0:(osd_handler.c:1883:osd_trans_dump_creds())   attr_set: 1/1/0, xattr_set: 2/15/0
Apr  3 01:29:08 foss01 kernel: Lustre: 12536:0:(osd_handler.c:1893:osd_trans_dump_creds())   write: 2/24182/0, punch: 0/0/0, quota 5/149/0
Apr  3 01:29:08 foss01 kernel: Lustre: 12536:0:(osd_handler.c:1900:osd_trans_dump_creds())   insert: 0/0/0, delete: 0/0/0
Apr  3 01:29:08 foss01 kernel: Lustre: 12536:0:(osd_handler.c:1907:osd_trans_dump_creds())   ref_add: 0/0/0, ref_del: 0/0/0

These are likely caused by patches from LU-14134 landing that changed the way transaction sizes are calculated (in theory reducing transaction size, but not always).

We might need an interface to check how extent per bytes going with system filled up, and also debug message to check how many credits calculated in  osd_declare_write_commit()

Issue Links

• is related to: LU-14134 reduce credits for new writing potentially

Comments

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" merged in patch https://review.whamcloud.com/45505/
Subject: LU-14641 osd-ldiskfs: write commit declaring improvement
Project: fs/lustre-release
Branch: b2_14
Current Patch Set:
Commit: 1734c4746d98dfa6fc6559841be8028a22465718

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/45505
Subject: LU-14641 osd-ldiskfs: write commit declaring improvement
Project: fs/lustre-release
Branch: b2_14
Current Patch Set: 1
Commit: 6db11a4e6e3214ac387433252986cd1850609260

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43446/
Subject: LU-14641 osd-ldiskfs: write commit declaring improvement
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0f81c5ae973bf7fba45b6ba7f9c5f4fb1f6eadcb

[Wang Shilong (Inactive)]

Current calculation is like:

 *raw_cpu_ptr(osd->od_extent_bytes_percpu) =

                DIV_ROUND_UP(old_bytes * (EXTENT_BYTES_DECAY -1) +

                             min(new_bytes, OSD_DEFAULT_EXTENT_BYTES),

                             EXTENT_BYTES_DECAY);

Yup, extent per bytes was limiting not exceeding 1M, this kind of calculation is more sensitive to fragments once there was, if

we tried to change it to new_bytes, extent_per_bytes might increase more quickly once we get one good large extent allocation.

 

I had no idea which once is better, maybe increase OSD_DEFAULT_EXTENT_BYTES to 4M might be enough?

 

[Andreas Dilger]

One thing I realized while running this on my home system is that osd_ldiskfs_map_write() is submitting the IO because of BIO size limits (1MB in my case), and not because the allocation is fragmented. That means that osd_extent_bytes() is always <= 1MB even when the extents are very large (128MB contiguous allocations in my case with a single-threaded writer, 8MB with multiple writers). It may be that we need to change how the extent stats are calculated?

[Andreas Dilger]

I'm wondering if I should try installing https://review.whamcloud.com/43232 "LU-14438 ldiskfs: improvements to mballoc" to see if that is improving the initial allocations or not, although this is already doing pretty good considering how little free space is available.