[LU-6254] Fix OFD/OSD prefetch for osd-ldiskfs and osd-zfs

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Major
• Reporter: Andreas Dilger
• Assignee: Alex Zhuravlev
• Resolution: Unresolved
• Labels: patch
• Severity: 3

Description

The ladvise code from LU-4931 (http://review.whamcloud.com/12458) and the recent ZFS performance improvements for read have highlighted that there are some inconsistencies in osd_read_prep() and osd_bufs_get_read() between osd-ldiskfs and osd-zfs.

It would be best for ladvise on ZFS if it were possible to just call dmu_prefetch() on the specified range and not have to actually set up the pages and drop them again immediately.

On ldiskfs, it appears that osd_read_prep() seems somewhat broken since it is removing pages from cache just before they are read:

        for (i = 0; i < npages; i++) {
                if (cache == 0)
                        generic_error_remove_page(inode->i_mapping,
                                                  lnb[i].lnb_page);
        }

which means that prefetching the page into OSS cache will be mostly useless, if it is discarded during the read phase immediately before reading it. I can understand dropping pages from cache for writes, but I'd think for reads it makes sense to leave them in cache if they are about to be used, especially if they are being explicitly prefetched.

Issue Links

• is related to: LU-5278 ZFS - many OST watchdogs with IOR
• is related to: LU-4931 New feature of giving server/storage ...
• is related to: LU-9069 ZFS sanity: FAIL: test_255a Speedup w...

Comments

[Bruno Faccini (Inactive)]

But concerning the pages unmap from cache in ldiskf:osd_read_prep(), I understand it will only occur if /proc/fs/lustre/osd-ldiskfs/<DEV>/read_cache_enable has been deliberately unset from its default value. Or did I miss something here?

[Andreas Dilger]

That is true, but I don't think it is the job of "read prep" to remove pages from cache if they are already IN cache. At most it should only remove pages from cache AFTER read. Otherwise, it isn't possible to prefetch pages into cache manually via ladvise(WILLNEED) when cache is disabled, nor is it possible to implement NOREUSE semantics.

[Gerrit Updater]

Li Xi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13996
Subject: LU-6254 ldiskfs: only remove page from cache after I/O
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6bec345c9517587ec1e9c6679fbdec2e47bb269c

[Li Xi (Inactive)]

As far as we test, the patch improves performance:

We see 15-20% improved for read performace and ~5% write performace dropped with OSS
READ/Write cache disabled after applying the patch, in generally, patch still makes sense for us.

So, please reconsider the patch again.

[Li Xi (Inactive)]

A cluster with 5000 clients were used to test this patch, and no problem was found.