[LU-10946] add an interface to load ldiskfs block bitmaps

• Type: Improvement
• Resolution: Won't Fix
• Priority: Minor
• Labels: patch
• Assignee: Wang Shilong (Inactive)
• Reporter: Wang Shilong (Inactive)

Description

During our benchmarking/testing, we found sometimes write performances are not stable enough and there are some small read during write which could drop thoughoutput of write performances.

It turned out that block bitmaps load make some latency here, also for a heavy fragment filesystem, we might need load many bitmaps to find some free blocks.

To improve above situation, we had a patch to load block bitmaps to memory and pin those bitmaps memory until unmount or we release the memory on purpose, this could stable write performances and improve performances of a heavy fragment filesystem.

Issue Links

• is related to: LU-17153 Random block allocation policy in ldiskfs
• is related to: LU-10967 MDT page cache management improvements
• is related to: LU-12970 improve mballoc for huge filesystems

Comments

[Nathan Dauchy (Inactive)]

For clarification on the metadata pre-loading done at NASA, I have uploaded the scripts.  They are run from cron like:

0 1   * * * root /usr/local/bin/read-meta.sh dump >> /var/log/lustre-read-meta.log 2>&1
*/15 * * * * root /usr/local/bin/read-meta.sh read >> /var/log/lustre-read-meta.log 2>&1

The original developer has left, but it sounds like the scripts were actually created with a focus on caching inode information (such as to speed up "ls -l"), not necessarily free blocks. Perhaps they have the nice side effect of refreshing the block bitmaps in cache, and if other changes in the last few years like flex_bg have improved inode table reading, then these scripts are now unnecessary?

[Nathan Dauchy (Inactive)]

Discussions of this at LUG brought up another issue, and possible solution... for cases of failover pairs, a server may have enough memory for pinning bitmaps in normal operation but OOM in a failover event with 2X the OST count.  To handle that case, and protect low-memory OSS in general, there could also be a configurable amount (80% by default?) of total memory threshold above which not to pin bitmaps.  I don't know if it would be better to pin as much as possible up to that threshold, or to pre-calculate whether all bitmaps for a given OST are pinnable based on total (or free) memory; but either way it should report a kernel error message if not able to pin without going over.

[Wang Shilong (Inactive)]

Hi Nathan Dauchy,

Thanks for your input here, looks you raised a lot of interesting question here. ^_

1)regarding memory requirement, yup, there will be memory pressure for pinning bitmaps.
just considering we have one block bitmap(4K) vs a 128M block group, it might eat a lot
of memory if system is big. and same requirement for inode bitmap. I guess system memory
might not be enough to pin both inode bitmap and block bitmaps.

2)we tried to just read bitmaps to memory and make it reclaimable but write performances
still not stable since those bitmap memory reclaimed easier before we want it.

3)I don't think it a good idea to load(pin) full inode table, since full inode table eat much more memory
than bitmaps..maybe some on-demand load inode table ahead makes sense, which I am not sure.

So I agreed we can make the patch more configurable with:
1)pre-read of bitmap to warm the cache without pinning.
2)option to pin bitmap

Thanks,
Shilong

[Nathan Dauchy (Inactive)]

This feature is of interest to NASA. (We currently use scripts around debugfs run periodically from cron to dump the bitmap information, and also the object trees with 'debugfs -c -R "ls O/0/d$i"'.)

I found this was discussed way back in LU-15, with memory requirement estimates (are they still valid?):
LU-15 comment-12883
Also from ticket LU-3631 is reading the inode bitmaps no longer really useful?

Regarding this patch, I think it would be helpful to include options to make it more generally configurable and usable for a wider variety of use-cases, not just pin block bitmaps. For example...

• Just do a pre-read of the bitmaps to warm the cache, without pinning.
• An option to either load at mount time, or load on demand.
• Differentiate between loading (and pinning) Data Block bitmap vs. Inode bitmap vs. the full Inode Table.

Thanks!

[Peter Jones]

ok - thanks wangshilong

 

[Wang Shilong (Inactive)]

I need cleanup our internal patch a bit, will push the patch to master very soon.