Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-3631

file stats are slow on filesystem start up

Details

    • Bug
    • Resolution: Not a Bug
    • Trivial
    • None
    • Lustre 1.8.9
    • None
    • 3
    • 9351

    Description

      NOAA had a question about an observed behavior. As part of their change management validation, they rerun a battery of tests after they make a change. One of those is mdtest. What has been observed is that after a downtime, the file stat performance is terrible. Like 400/s. After a while, it goes up to around 10-15k. It seems to be a function of load as opposed to time. The more the filesystem is used (due to other testing), the faster the stat performance increases.

      Are there any thoughts on why this might happen? I have tried preloading the OST metadata, but that didn't seem to have any effect. I thought that IB routing might be an issue, but even when the IB fabric is untouched, we see this issue. It seems like it must be a cache issue, but I am unsure what caches are being warmed up. Any insight on why we are seeing this and how to preload the cache would be great.

      Thanks.

      Attachments

        Issue Links

          Activity

            [LU-3631] file stats are slow on filesystem start up

            Ok great, thanks!

            kitwestneat Kit Westneat (Inactive) added a comment - Ok great, thanks!

            This is a fairly typical scenario as the filesystem cache is warmed up after a restart. In particular, the inode bitmaps need to be read from disk. If there are partially allocated itable blocks are also read from disk at allocation time (unallocated blocks are not read from disk).

            The inode bitmaps can be prefetched at boot time by running:

            dumpe2fs /dev/mdsdev 2> /dev/null
            

            The problem of loading the inode bitmaps at startup time is avoided with Lustre 2.x formatted filesystems by using the flexbg feature.

            adilger Andreas Dilger added a comment - This is a fairly typical scenario as the filesystem cache is warmed up after a restart. In particular, the inode bitmaps need to be read from disk. If there are partially allocated itable blocks are also read from disk at allocation time (unallocated blocks are not read from disk). The inode bitmaps can be prefetched at boot time by running: dumpe2fs /dev/mdsdev 2> /dev/null The problem of loading the inode bitmaps at startup time is avoided with Lustre 2.x formatted filesystems by using the flexbg feature.
            pjones Peter Jones added a comment -

            Oleg

            What do you recommend?

            Peter

            pjones Peter Jones added a comment - Oleg What do you recommend? Peter

            People

              green Oleg Drokin
              kitwestneat Kit Westneat (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: