[LU-9232] Reduce osd-ldiskfs page lock hold time for read

Duplicate — Thu, 11 Jun 2020 09:23:31 +0000

It seems that there is spinlock contention in osd_get_bufs->osd_get_page->find_or_create_page(), when many threads are reading the same offset of one object (executable load from Lustre) and I'm wondering if we can optimize this somehow?

Firstly, we should consider fetching multiple pages at one time. That will reduce locking on the address space and hopefully make page lookup more efficient. One option is to use mapping->readpages() (used to be mpage_readpages() but has changed to ext4_mpage_readpages() in newer kernels to handle encrypted files), or fop->read_iter() (ext4_file_read_iter(), need to make a fake file struct with f_mapping set, and set O_NOATIME and disable readahead to avoid a messy callpath in the VFS) and just convert the niobuf to an iov directly. This has the possibility of removing a lot of code from osd-ldiskfs that was created before ext4 used bio-based read/write.

The other potential major win is if we had a fast path for reads where all of the pages are already in cache, so we don't have to hold the page locks over the whole RPC, blocking the other threads from accessing the same data? Using find_get_page() appears to avoid the page lock completely, since it is only needed when reading the page from disk? In the case of a fully cached object, we could avoid all contention. We would only fall back to locking the page if actual disk IO is needed. This is largely already encapsulated in the VFS/MM read path, so if we could use that it would save a lot of complexity.

The current page locking was done for simplicity mostly - so that we unlock pages unconditionally in osd_bufs_put(). It should be possible to unlock uptodate pages in osd_read_prep() and fix osd_bufs_put() to skip unlock if it's called with rw=0. Something similar was done in Lustre 1.8 to drop page locks on read.

Whamcloud Community JIRA

[LU-9232] Reduce osd-ldiskfs page lock hold time for read