Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-6254

Fix OFD/OSD prefetch for osd-ldiskfs and osd-zfs

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • Lustre 2.8.0
    • 3
    • 17511

    Description

      The ladvise code from LU-4931 (http://review.whamcloud.com/12458) and the recent ZFS performance improvements for read have highlighted that there are some inconsistencies in osd_read_prep() and osd_bufs_get_read() between osd-ldiskfs and osd-zfs.

      It would be best for ladvise on ZFS if it were possible to just call dmu_prefetch() on the specified range and not have to actually set up the pages and drop them again immediately.

      On ldiskfs, it appears that osd_read_prep() seems somewhat broken since it is removing pages from cache just before they are read:

              for (i = 0; i < npages; i++) {
                      if (cache == 0)
                              generic_error_remove_page(inode->i_mapping,
                                                        lnb[i].lnb_page);
              }
      

      which means that prefetching the page into OSS cache will be mostly useless, if it is discarded during the read phase immediately before reading it. I can understand dropping pages from cache for writes, but I'd think for reads it makes sense to leave them in cache if they are about to be used, especially if they are being explicitly prefetched.

      Attachments

        Issue Links

          Activity

            [LU-6254] Fix OFD/OSD prefetch for osd-ldiskfs and osd-zfs

            A cluster with 5000 clients were used to test this patch, and no problem was found.

            lixi Li Xi (Inactive) added a comment - A cluster with 5000 clients were used to test this patch, and no problem was found.

            As far as we test, the patch improves performance:

            We see 15-20% improved for read performace and ~5% write performace dropped with OSS
            READ/Write cache disabled after applying the patch, in generally, patch still makes sense for us.

            So, please reconsider the patch again.

            lixi Li Xi (Inactive) added a comment - As far as we test, the patch improves performance: We see 15-20% improved for read performace and ~5% write performace dropped with OSS READ/Write cache disabled after applying the patch, in generally, patch still makes sense for us. So, please reconsider the patch again.

            Li Xi (lixi@ddn.com) uploaded a new patch: http://review.whamcloud.com/13996
            Subject: LU-6254 ldiskfs: only remove page from cache after I/O
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 6bec345c9517587ec1e9c6679fbdec2e47bb269c

            gerrit Gerrit Updater added a comment - Li Xi (lixi@ddn.com) uploaded a new patch: http://review.whamcloud.com/13996 Subject: LU-6254 ldiskfs: only remove page from cache after I/O Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 6bec345c9517587ec1e9c6679fbdec2e47bb269c

            That is true, but I don't think it is the job of "read prep" to remove pages from cache if they are already IN cache. At most it should only remove pages from cache AFTER read. Otherwise, it isn't possible to prefetch pages into cache manually via ladvise(WILLNEED) when cache is disabled, nor is it possible to implement NOREUSE semantics.

            adilger Andreas Dilger added a comment - That is true, but I don't think it is the job of "read prep" to remove pages from cache if they are already IN cache. At most it should only remove pages from cache AFTER read. Otherwise, it isn't possible to prefetch pages into cache manually via ladvise(WILLNEED) when cache is disabled, nor is it possible to implement NOREUSE semantics.

            But concerning the pages unmap from cache in ldiskf:osd_read_prep(), I understand it will only occur if /proc/fs/lustre/osd-ldiskfs/<DEV>/read_cache_enable has been deliberately unset from its default value. Or did I miss something here?

            bfaccini Bruno Faccini (Inactive) added a comment - But concerning the pages unmap from cache in ldiskf:osd_read_prep(), I understand it will only occur if /proc/fs/lustre/osd-ldiskfs/<DEV>/read_cache_enable has been deliberately unset from its default value. Or did I miss something here?

            People

              bzzz Alex Zhuravlev
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: