[LU-3631] file stats are slow on filesystem start up

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 1.8.9
• Type: Bug
• Priority: Trivial
• Reporter: Kit Westneat (Inactive)
• Assignee: Oleg Drokin
• Resolution: Not a Bug
• Severity: 3

Description

NOAA had a question about an observed behavior. As part of their change management validation, they rerun a battery of tests after they make a change. One of those is mdtest. What has been observed is that after a downtime, the file stat performance is terrible. Like 400/s. After a while, it goes up to around 10-15k. It seems to be a function of load as opposed to time. The more the filesystem is used (due to other testing), the faster the stat performance increases.

Are there any thoughts on why this might happen? I have tried preloading the OST metadata, but that didn't seem to have any effect. I thought that IB routing might be an issue, but even when the IB fabric is untouched, we see this issue. It seems like it must be a cache issue, but I am unsure what caches are being warmed up. Any insight on why we are seeing this and how to preload the cache would be great.

Thanks.

Issue Links

• is related to: LU-10967 MDT page cache management improvements

Comments

[Peter Jones]

Oleg

What do you recommend?

Peter

[Andreas Dilger]

This is a fairly typical scenario as the filesystem cache is warmed up after a restart. In particular, the inode bitmaps need to be read from disk. If there are partially allocated itable blocks are also read from disk at allocation time (unallocated blocks are not read from disk).

The inode bitmaps can be prefetched at boot time by running:

dumpe2fs /dev/mdsdev 2> /dev/null

The problem of loading the inode bitmaps at startup time is avoided with Lustre 2.x formatted filesystems by using the flexbg feature.

[Kit Westneat (Inactive)]

Ok great, thanks!